From old wiki http://en.gentoo-wiki.com/wiki/Squashed_Portage_Tree

This article describes how to reduce the Portage tree, using a squashfs image, to 30-40MB. A good alternative to having the portage tree on a separate partition. For a list of publicly available copies please see the bottom of this article.

Prerequisites¶

Required Packages¶

You will need sys-fs/squashfs-tools which provides mksquashfs,

emerge -av squashfs-tools

And of course the aufs module:

emerge -av aufs3

Note: If you are using kernel 2.6 then emerge sys-fs/aufs2 instead. In case aufs2 complains about missing Symbols related to inotify, you can remove inotify-useflag from aufs as a workaround.

Note: Alternative you can use the “live” ebuild, aufs2-9999 or a kernel source package that includes aufs, like sys-kernel/zen-sources.

Kernel Configuration¶

Kernels based on sys-kernel/gentoo-sources include SquashFS by default.

# Linux Kernel Configuration: SquashFS Kernel Config
Device Drivers --->
  Block Devices --->
    <M> Loopback device support
File systems --->
  Miscellaneous Filesystems --->
    <M> SquashFS

It is of course possible to have these built-in, but we avoid a reboot when building these as modules.

Single tree¶

Preparation¶

For a single squashed tree, you will need to move anything not part of the portage tree out of PORTDIR( /usr/portage ). This usally includes DISTDIR and laymans storage. Modify /etc/make.conf,

# File: /etc/make.conf
...
DISTDIR="/var/portage/distfiles"
...

Remove /usr/portage/distfiles,

rm /usr/portage/distfiles -rf

Note: The following step is only needed if you are using a layman version older than 1.2.0

Layman’s configuration file is /etc/layman/layman.cfg. Here we move layman’s storage directory to /var/portage, same directory as DISTDIR,

# File: /etc/layman/layman.cfg
...
storage   : /var/portage/layman
...

Move the layman storage directory to our new location,

mv /usr/portage/local/layman /var/portage/layman

init.d/squash_portage¶

Paste this into /etc/init.d/squash_portage,

File: /etc/init.d/squash_portage
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# /etc/init.d/squash_portage allows efficient compression of
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs and sys-fs/aufs are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin@users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)
# 2009-02-24, v.0.1.6(1) Weedy <weedy2887@gmail.com>
# 2009-03-20, v.0.1.7(1) j0inty <j0inty@stollfuss.net>
# 2009-07-10, v.0.1.8(1) j0inty
# 2009-09-01. v.0.1.9(1) nall <soir@fuzzysock.net>

extra_started_commands="sync"

source /usr/share/portage/config/make.globals
source /etc/make.conf
SQFS_CUR="$SQFS_DIRNAME/portage.sqfs"
SQFS_NEW="$SQFS_DIRNAME/portage-current.sqfs"
SQFS_OLD="$SQFS_DIRNAME/portage-old.sqfs"
DEF_RW="/dev/shm/.portage-rw"
SQFS_OPTS="-force-uid portage -force-gid portage -no-duplicates"

depend() {
    need localmount modules
}


check_support() {
    if ! [ -x /usr/bin/mksquashfs ] ; then
        eerror "ERROR: sys-fs/squashfs-tools is not installed."
        return 1
    fi
    if ! [ -w /dev/loop0 ] ; then
        eerror "ERROR: loopback support is not available."
        return 1
    fi
    if ! [[ $(grep -s $'\taufs$' /proc/filesystems) ]] ; then
        eerror "ERROR: aufs filesystem support is not available."
        return 1
    fi
    if ! [[ $(grep -s $'\tsquashfs$' /proc/filesystems) ]] ; then
        eerror "ERROR: squashfs filesystem support is not available."
        return 1
    fi
    return 0
}

makeImage() {
    mksquashfs $PORTDIR $SQFS_NEW $SQFS_OPTS # 2>/dev/null
    retval=$?
    ln -sf $SQFS_NEW $SQFS_CUR
    eend $retval
}

sync() {
    ebegin "Syncing portage tree"
    eval $SYNC_CMDS
    #svc_stop; svc_start
    stop
    start
    eend 0
}

start() {
    check_support || return 1
    if [ -f "$SQFS_CUR" ]; then
        ebegin "SQFS-PORTAGE: Mounting read-only squashfs image"
        mount -rt squashfs -o loop,nodev,noexec $SQFS_CUR $PORTDIR
        retval=$?
        [ $retval -ne 0 ] && return $retval
    else
        if [ ! -f "/usr/portage/metadata/timestamp.chk" ]; then
            ebegin "SQFS-PORTAGE: $PORTDIR looks empty or corrupted, syncing"
            eval $SYNC_CMDS
        fi
        einfo "  $SQFS_CUR does not exist, creating"
        mkdir -p $SQFS_DIRNAME
        makeImage
        [ $? -ne 0 ] && eerror "ERROR: failed to create initial tree image"
        einfo "Clearing ${PORTDIR}"
        rm -r ${PORTDIR}
        mkdir ${PORTDIR}
        start
        eend 0
    fi

    ebegin "Mounting read-write with aufs"
    if [ ! $PORTAGE_RW ] ; then
        einfo "  mounted in tmpfs (RAM)"
        PORTAGE_RW="${DEF_RW}"
    fi
    [ -d $PORTAGE_RW ] || mkdir -p $PORTAGE_RW
    chmod 0750 $PORTAGE_RW
    chown portage:portage $PORTAGE_RW
    mount -t aufs -o nodev,noexec,br=$PORTAGE_RW=rw:$PORTDIR=ro aufs $PORTDIR
    eend $?

    if [ "$DISTDIR" == "/usr/portage/distfiles" ]; then
        mkdir -p /usr/local/distfiles 
        mount -o bind /usr/local/distfiles /usr/portage/distfiles
        ewarn "DISTDIR is currently inside the portage tree. It has been bind 
        mounted to keep the SquashFS image small."
    fi
}

stop() {
    ebegin "SQFS-PORTAGE: Stopping and unmounting"
    [ ! $PORTAGE_RW ] && PORTAGE_RW="${DEF_RW}"
    if [ $(du -s --exclude=.w* $PORTAGE_RW | cut -f 1) -gt 4 ]; then
        einfo "  Changes detected, updating image."
        mv -f $SQFS_NEW $SQFS_OLD
        makeImage
        rm -f $SQFS_OLD
    else
        einfo "  No changes detected, skipping update."
        eend 0
    fi

    if [ "$DISTDIR" == "/usr/portage/distfiles" ]; then
        einfo "  Unmounting distfiles"
        umount /usr/local/distfiles
    fi;

    einfo "  Unmounting the tree"
    umount -t aufs  $PORTDIR
    umount -t squashfs $PORTDIR
    rm -rf $PORTAGE_RW
    eend 0
}

Make it executable,

chmod 755 /etc/init.d/squash_portage

conf.d/squash_portage¶

Paste this in the corresponding configuration file /etc/conf.d/squash_portage,

# File: /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

# SQFS_DIRNAME points to the directory that will contain the sqfs
# images, recommended value is /var/portage
SQFS_DIRNAME="/var/portage"

# Leave PORTAGE_RW empty for use with tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
PORTAGE_RW=""

# If you need more then just emerge --sync, or are using another
# package manager add them here. Example SYNC_CMDS="/usr/bin/layman -S; /usr/bin/eix-sync"
SYNC_CMDS="emerge --sync"

Usage¶

To make sure you always have a mounted portage tree:

rc-update add squash_portage default

Now it is time to initialize the tree. Simply start the service.

/etc/init.d/squash_portage start

Multiple trees¶

Note: The following option is only recommended if you know exactly what you intend to do.

Preparation¶

For multiple squashed trees, you probably want to move DISTDIR (usually /usr/portage/distfiles) out of PORTDIR (usually /usr/portage), or else it’ll take a very long time to create a portage sqfs image, without any significant size reduction.

What we’ll describe here is an example configuration, for three trees, each of located in /var/portage/{portage,layman,local}, with it’s original locations being /usr/portage, /usr/local/portage/layman and /usr/local/portage. One could achieve the same with only two trees by merging local with layman. But this would side effect you dependent of layman for your local tree, and our script configuration, to work without modifications.

First we make sure that /var/portage and sub-dirs does exist.

mkdir -p /var/portage/{portage,layman,local}

Remember that we want every tree in /var/portage, and, we also need layman to write it’s overlays to /etc/make.overlays, instead of /usr/local/layman/make.conf. So, modify /etc/make.conf,

Note: Remember to remove the old “source /usr/local/portage/layman/make.conf” before entering the new one

# File: /etc/make.conf
source /etc/make.overlays
...
PORTDIR=/var/portage/portage
LOCALDIR="/var/portage/local"
DISTDIR="/var/portage/distfiles"
...

and /etc/layman/layman.cfg,

# File: /etc/layman/layman.cfg
...
storage   : /var/portage/layman
...
make_conf : /etc/make.overlays
...

Move /usr/portage/distfiles,

mv /usr/portage/distfiles /var/portage/distfiles

Or remove /usr/portage/distfiles,

rm /usr/portage/distfiles -rf

Move /usr/local/layman/make.conf,

mv /usr/local/layman/make.conf /etc/make.overlays

Modify /etc/make.overlays overlays to /var/portage/,

Note: Example configuration, you have to adapt your configuration

File: /etc/make.overlays
PORTDIR_OVERLAY="
/var/portage/layman/sunrise
/var/portage/layman/gentoo-china
/var/portage/layman/devnull
/var/portage/layman/kde-testing
/var/portage/layman/mozilla
/var/portage/layman/qting-edge
$PORTDIR_OVERLAY
/var/portage/local"

Before we create the images we need to split layman from local, doing:

mv /usr/local/portage/layman/ /usr/local/layman/

Now we have to create the sqfs images, since our script expect the images to be at /var/portage/, named as sqfs.${TREE}-current.sqfs, we proceed as following:

mksquashfs /usr/portage/ sqfs.portage-current.sqfs
mksquashfs /usr/local/portage/ sqfs.local-current.sqfs
mksquashfs /usr/local/layman/ sqfs.layman-current.sqfs

Since some system files need files at /usr/portage/, we have to symlink after we remove them.
It’s safer to move:

mv /usr/portage/eclass /usr/portage/eclass.old
mv /usr/portage/profiles /usr/portage/profiles.old

After we symlink:

ln -fs /var/portage/portage/eclass/ /usr/portage/eclass
ln -fs /var/portage/portage/profiles/ /usr/portage/profiles

Init script¶

You can now create and configure /etc/init.d/squash_portage, first of all create it:

File: /etc/init.d/squash_portage
#!/sbin/runscript
# Copyright 1999-2006 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $
#
# /etc/init.d/squash_portage allows efficient compression of
# Gentoo portage arborescence
#
# It requires support for the loop device and squashfs enabled in the kernel,
# module autoloading is also *highly* recommended.
# sys-fs/squashfs, sys-fs/squashfs-tools and sys-fs/aufs
# or sys-fs/aufs2 (recommended) are necessary for read-write support.
#
# Author: Mathias Laurin <mathias_laurin@users.sourceforge.net>
# 2006-11-28, v.0.1.5(4)
# 2009-02-24, v.0.1.6(1) Weedy <weedy2887@gmail.com>
# 2009-03-20, v.0.1.7(1) j0inty <j0inty@stollfuss.net>
# 2009-07-10, v.0.1.8(1) j0inty
# 2009-08-14, v.0.1.9(1) morris <mauricioarozi@gmail.com>
# 2009-09-01. v.0.1.9(1) nall <soir@fuzzysock.net>
# 2009-11-22, v.0.2.0(3) morris <mauricioarozi@gmail.com>

opts="sync"

depend() {
    need localmount modules
}

retuval() {
    retval=${1}
    eend ${retval}
    [ ${retval} -ne 0 ] && \
    ewarn "${2}" && \
    return 1
}

check_support() {
    if ! [ -x /usr/bin/mksquashfs ] ; then
        eerror "ERROR: sys-fs/squashfs-tools is not installed."
        return 1
    fi
    if ! [ -w /dev/loop0 ] ; then
        eerror "ERROR: loopback support is not available."
        return 1
    fi
    if ! [[ $(grep -s aufs /proc/filesystems) ]] ; then
        eerror "ERROR: aufs filesystem support is not available."
        return 1
    fi
    if ! [[ $(grep -s squashfs /proc/filesystems) ]] ; then
        eerror "ERROR: squashfs filesystem support is not available."
        return 1
    fi
}


sync() {
    source /etc/make.globals        # MUST source inside function, causes initramfs errors otherwise
    source /etc/make.conf
    ebegin "Syncing portage tree"
    if [ ${SYNC_CMD} ]; then
        eval ${SYNC_CMD}
    else
        [ -x '/usr/bin/emerge' ] && local SYNC_CMD='command emerge --sync'
        [ -x '/usr/bin/paludis' ] && local SYNC_CMD='command paludis --sync'
        [ -x '/usr/bin/eix' ] && local SYNC_CMD='command eix-sync'
        # make eix-sync work with layman overlays shound't be default
        # [ -x '/usr/bin/eix' ] && [ -x '/usr/bin/layman' ] && [ ! -f /etc/eix-sync.conf ] && `/bin/echo '*' > /etc/eix-sync.conf`

        eval "${SYNC_CMD}"
    fi
    retuval ${?} "Error: ${SYNC_CMD}"
    stop
    start
    eend 0
}

start() {
    source /etc/make.globals    # MUST source inside function
    source /etc/make.conf

    ebegin "Mounting read-only squashfs image(s)"
    check_support
    if [ ! -d "${SQFS_DIRNAME}" ]; then
        einfo "${SQFS_DIRNAME} does not exist, creating"
        mkdir -p "${SQFS_DIRNAME}"
        retuval ${?} "Error: mkdir -p ${SQFS_DIRNAME}"
    fi

    for i in ${SQFSS[@]}; do
        einfo "Mounting ${i}"
        mount -rt squashfs -o loop,nodev,noexec "${SQFS_DIRNAME}/sqfs.${i}-current.sqfs" "${SQFS_DIRNAME}/${i}"
        retuval ${?} "Error: mount -rt squashfs -o loop,nodev,noexec \"${SQFS_DIRNAME}/sqfs.${i}-current.sqfs\" \"${SQFS_DIRNAME}/${i}\""
        [ "${SQFS_DIST}" ] || \
        if [ `echo ${DISTDIR} | grep "${SQFS_DIRNAME}/${i}"` ]; then
            mkdir -p /usr/local/distfiles
            retuval ${?} "Error: mkdir -p /usr/local/distfiles"
            mount -o bind "/usr/local/distfiles" "${DISTDIR}"
            retuval ${?} "Error: mount -o bind /usr/local/distfiles ${DISTDIR}"
            ewarn "DISTDIR is currently inside of ${SQFS_DIRNAME}/${i} tree. 
            It has been bind mounted to keep the SquashFS image small."
        fi
    done; unset i rw

    einfo "Mounting read-write with aufs"
    for i in `seq 0 $[${#FSRW[@]}-1]`; do
        local RW=${FSRW[${i}]:-"/dev/shm/.${SQFSS[${i}]}-rw"}
        [ -d "${RW}" ] || einfo "Creating ${RW}" && mkdir -p "${RW}"
        retuval ${?} "Error: mkdir -p \"${RW}\"" 
        chmod 0750 "${RW}"
        retuval ${?} "Error: chmod 0750 \"${RW}\""
        chown portage:portage "${RW}"
        retuval ${?} "Error: chown portage:portage \"${RW}\""
    done; unset a i rw

    for i in ${SQFSS[@]}; do
        [ ${a} ] && a=$[${a}+1] || local a=0
        local RW=${FSRW[${a}]:-"/dev/shm/.${i}-rw"}
        einfo "${RW}"
        mount -t aufs -o "nodev,noexec,br:${RW}=rw:${SQFS_DIRNAME}/${i}=ro" aufs "${SQFS_DIRNAME}/${i}"
        retuval ${?} "Error: mount -t aufs -o \"nodev,noexec,br:${RW}=rw:${SQFS_DIRNAME}/${i}=ro\" aufs \"${SQFS_DIRNAME}/${i}\""
        einfo "${SQFSS[${a}]} mounted in ${RW}"
    done; unset a i rw
    eend ${?}
}

stop() {
    source /etc/make.globals    # MUST source inside function
    source /etc/make.conf

    check_support
    if [ "$RC_RUNLEVEL" != shutdown ]; then # OpenRC timeout doesn't allow this kind of thing
        ebegin "Updating portage tree"
        for i in `seq 0 $[${#FSRW[@]}-1]`; do
            local RW=${FSRW[${i}]:-"/dev/shm/.${SQFSS[${i}]}-rw"}
            einfo "Syncing the tree ${SQFSS[${i}]}"
            if [ ! -z "`/bin/ls -A "${RW}" | /bin/grep -v .wh.`" ]; then
                einfo "Syncing..."
                local SOLD="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}-old.sqfs"
                local SNEW="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}-current.sqfs"
                local SS="${SQFS_DIRNAME}/sqfs.${SQFSS[${i}]}.sqfs"
                mv -f "${SNEW}" "${SOLD}"
                retuval ${?} "Error: mv -f \"${SNEW}\" \"${SOLD}\""
                [ -w "${SQFS_DIRNAME}" ] && \
                /usr/bin/mksquashfs "${SQFS_DIRNAME}/${SQFSS[${i}]}" "${SNEW}" ${SQFS_OPTS}
                retuval ${?} "Error: /usr/bin/mksquashfs \"${SQFS_DIRNAME}/${SQFSS[${i}]}\" \"${SNEW}\" ${SQFS_OPTS}"
                /bin/ln -fs "${SNEW}" "${SS}"
                retuval ${?} "Error: /bin/ln -fs \"${SNEW}\" \"${SS}\""
            else
                einfo "Nothing to do"
            fi
        done; unset a i retval
    fi

    ebegin "Unmounting the tree(s)"
        for i in ${SQFSS[@]}; do
        [ ${a} ] && a=$[${a}+1] || local a=0
        local RW=${FSRW[${a}]:-"/dev/shm/.${i}-rw"}
                umount -f -l -t aufs "${SQFS_DIRNAME}/${i}"
        retuval ${?} "Error: umount -f -l -t aufs \"${SQFS_DIRNAME}/${i}\""
        umount -f -l -t squashfs "${SQFS_DIRNAME}/${i}"
        retuval ${?} "Error: umount -f -l -t squashfs \"${SQFS_DIRNAME}/${i}\""
        /bin/rm -fr "${RW}"
        retuval ${?} "Error: /bin/rm -fr \"${RW}\""
        done; unset i retval

    eend 0
}

Make it executable,

chmod 755 /etc/init.d/squash_portage

Now to configure, create and edit the corresponding configuration file /etc/conf.d/squash_portage,

Note: Configuration file for example setup, remember to change it!

# File: /etc/conf.d/squash_portage
# /etc/conf.d/squash_portage

# Fill SQFS_DIST with anything if you don't mind to squash your distfiles
# It is recommended to leave blank
SQFS_DIST=""

# SQFS_DIRNAME points to the directory that will contain the sqfs images
SQFS_DIRNAME="/var/portage"

# Options used for ALL image creations
# Do not change unless you know what you are doing
SQFS_OPTS="-force-uid portage -force-gid portage -no-duplicates"

# Populate FSRW array with nulls ("", '', 0...) to use tmpfs, a ram-based filesystem,
# This is recommended unless you are short of RAM
FSRW=('' '' '')

# FS mount array, used in substitution: ${SQFS_DIRNAME}/${SQFSS[@]}
SQFSS=("portage" "local" "layman")

# If you need more than just emerge --sync, or are using another
# package manager add them here. Example SYNC_CMDS="/usr/bin/layman -S; /usr/bin/eix-sync"
SYNC_CMD="eix-sync"
# Note: eix-sync can run `layman -S` automatically if you run `echo '*' > /etc/eix-sync.conf` once

Check your initial setup running the script once.

/etc/init.d/squash_portage start

Now complete your setup restarting it, so it will create necessary symlinks while stopping.

/etc/init.d/squash_portage restart

To make sure you always have a mounted portage tree:

rc-update add squash_portage default

If everything went alright, you can now delete the all old trees content but /usr/portage/eclass and /usr/portage/profiles links from /usr/ if you want.

Usage¶

You can sync the trees with:

/etc/init.d/squash_portage sync

You can also start, stop, and restart it:

/etc/init.d/squash_portage start
/etc/init.d/squash_portage stop
/etc/init.d/squash_portage restart

Note: Restart is useful if you synced trees manually, and want to sync just the sqfs images automatically

Just another weblog about Linux

Squashed portage tree

Prerequisites¶

Required Packages¶

Kernel Configuration¶

Single tree¶

Preparation¶

init.d/squash_portage¶

conf.d/squash_portage¶

Usage¶

Multiple trees¶

Preparation¶

Init script¶

Usage¶

Comments

Pages

Categories

Tags