git-annex

2016-05-24

blog

linux / git

Table of Contents

Create a repository

cd ~
git clone git@gitlab.com:rychly/git-annex.git \
|| ( mkdir git-annex && git --git-dir=git-annex init )
cd git-annex
git annex init --version=6

Configure the repository

git annex describe <name> <description> – changes the description of the repository
git annex wanted <name> [expression] – gets/sets preferred content expression in this repository
git annex group <name> <groupname> – adds the repository to a group (e.g., into one of standard groups)
git annex schedule <name> [expression] – gets/sets scheduled jobs (expression is “fsck self” or “fsck UUID” action, where UUID is the UUID of a remote to fsck, followed by the duration to allow the action to run, and finally the schedule of when to run it)
git annex trust <name> – the repository is trusted to not unexpectedly lose content
git annex untrust <name> – the repository is not trusted and could lose content at any time
git config remote.<name>.annex-cost <double> – sets cost of annexed files transfers, repos with lower costs are preferred (defaults are 100 for local repositories and 200 for remote repositories)
git config remote.<name>.annex-cost-command <command> – the command is run, and the number it outputs is used as the cost
git config remote.<name>.annex-ignore true – prevents git-annex from storing (syncing) file contents on this remote by default (–from and –to options still work)
git config remote.<name>.annex-sync false – prevents git-annex sync (and the git-annex assistant) from syncing with this remote
git config annex.ssh-options '-o ForwardX11=no' – does not attempt forward X11 requests when using ssh to talk to all remotes

Set basic remotes in the repository

cd ~/git-annex
## disable X11 forwarding in ssh connections
git config annex.ssh-options '-o ForwardX11=no'
## require more than one copy of a file exists
git annex numcopies 2
## set local repository
git annex group . 'client'
git annex wanted . 'standard'
git annex schedule . 'fsck self 10m every day at any time'
## set GitLab.com LFS to enable a transfer mode
git annex describe origin GitLab.com
git annex wanted origin 'standard'
git annex group origin 'transfer'

Set an encrypted SSH remote bare Git repository as a backup storage

${USER} is the current username (both for the local and the remote)
${REMOTE_HOST} is the remote host-name
${REMOTE_REPO} is the remote repository directory-name
${GPG_KEY_ID} is a gpg key id, ususally an email

Remote

## install git-annex
sudo yum install git-annex -y
## initialize a bare Git repository
git init --bare /home/${USER}/${REMOTE_REPO}.git

Local

## install git-remote-gcrypt
sudo emerge -av dev-vcs/git-remote-gcrypt
## get a gpg key id
gpg --list-keys || gpg --gen-key
## enabel or add gcrypt remote in a backup mode
git annex enableremote ${REMOTE_HOST} \
|| git annex initremote ${REMOTE_HOST} type=gcrypt \
 gitrepo=ssh://${USER}@${REMOTE_HOST}/home/${USER}/${REMOTE_REPO}.git \
 keyid=${GPG_KEY_ID}
git annex describe ${REMOTE_HOST} "gcrypt:${USER}@${REMOTE_HOST}:~/${REMOTE_REPO}.git"
git annex wanted ${REMOTE_HOST} 'include=*'
git annex group ${REMOTE_HOST} 'backup'
git config remote.${REMOTE_HOST}.annex-cost 150

Set Mega.nz remote as a backup storage

## add/enable Mega.nz remote in a backup mode
USERNAME="USER" PASSWORD="PASSWORD" ( git annex enableremote mega \
|| git annex initremote -d mega \
type=external externaltype=mega encryption=none folder=git-annex )
git annex describe mega 'Mega.nz'
git annex wanted mega 'include=*'
git annex group mega 'backup'
git config remote.mega.annex-cost 300
git config remote.mega.annex-ignore true

Enable synchronization requests via Jabber/XMPP

## add XMPP remote
echo 'XMPPCreds {xmppUsername = "USER", xmppPassword = "PASSWORD",\
 xmppHostname = "xmpp.l.google.com.", xmppPort = 5222,\
 xmppJID = "USER@gmail.com"}' > .git/annex/creds/xmpp
chmod 600 .git/annex/creds/xmpp
# and configure it via webapp in Configuration / Jabber account

Sample `.git/config` file

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[annex]
	uuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
	version = 6
	ssh-options = -o ForwardX11=no
[filter "annex"]
	smudge = git-annex smudge %f
	clean = git-annex smudge --clean %f
[branch "master"]
	remote = origin
	merge = refs/heads/master
[remote "origin"]
	url = git@gitlab.com:gitlab-user/repo-name.git
	fetch = +refs/heads/*:refs/remotes/origin/*
	annex-uuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
[remote "mega"]
	annex-externaltype = mega
	annex-uuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
	annex-availability = GloballyAvailable
	annex-cost = 300
	annex-ignore = true
[remote "${REMOTE_HOST}"]
	url = gcrypt::ssh://${USER}@${REMOTE_HOST}/home/${USER}/${REMOTE_REPO}.git
	fetch = +refs/heads/*:refs/remotes/gort/*
	gcrypt-participants = XXXXXXXXXXXXXXXX
	gcrypt-signingkey = XXXXXXXXXXXXXXXX
	gcrypt-publish-participants = true
	gcrypt-id = :id:xxxxxxxxxxxxxxxxxxxx
	annex-gcrypt = shell
	annex-uuid = xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
	annex-cost = 150

Version 6 (locked and unlocked files, annex.thin)

In a v6 repositoring, the locking mode of files is stored and synchronized through repositories.

`git annex add <files>`

Add files into the repository (in recursive mode, it will skip dotfiles, see git add). If annex.addunlocked is true and it is in a v6 respository, it keeps the files in unlocked mode, otherwise the files will be switched to locked mode.

To unlock all annexed files and always add files unlocked mode (i.e., to keep the unlocked), run:

git config annex.addunlocked true
git annex adjust --unlock

Unlocked files are more difficult to synchronize (to check for modifications and to synchronize throught multiple remotes).

`git add <files>`

In a v6 respository, it adds files into the repository and keeps them in unlocked mode.

In a pre-v6 resporitory, or if the added files are dot-files (i.e., start with “.” character), the files are added directly into a git repository, not into the git-annex repository.

To move files from a git repository to a git-annex repository, run:

find <items> -type f ! -name '.git*' \
-exec sh -c 'git rm --cached {} && git annex add {} -v' \;

`git annex unlock <files>`

Unlock files, i.e., replace symlinks to git-annex objects (that are read-only) with regular files (writable).

In a v6 repository with annex.thin set to true, these unlocked (regular) files are hard-links to the git-annex objects that are set as writable. In this case, modifications of the files' content are rewriting the original content of the files which is lost, and if needed, it must be loaded from another repository.
In a pre-v6 repository or with annex.thin set to false, the unlocked (regular) files are detached copies of the git-annex objects. In this case, modifications of the files' content are isolated and do not affect the original conent that is available in the local repository.

To enable annex.thin, run:

## avoid the overhead of the second local copy of unlocked files
## (there must be a backup repository, i.e., more than one existing online copy of files is required)
git config annex.thin true
git annex fix

`git annex lock <files> --force`

Revert any modifications in unlocked files and lock them, i.e., replace their regular files (writable) with symlinks to git-annex objects (that are read-only).

In annex.thin enabled repository, it is not possible to revert the modifications by locking the files without loosing the files' content (see git annex unlock). The original (pre-modification) content is not available in the local repository and must be obtained from another repository (which is usualy not possible to do offline).

Keep the repository up to date automatically

In a v6 respository (see the smudge mode), git-annex assistant forces the repository into the deprecated direct mode and does not use new adjusted branches feature.

mkdir -p ~/.config/git-annex
echo "${HOME}/git-annex" > ~/.config/git-annex/autostart
git-annex assistant --autostop
git-annex assistant --autostart --notify-start --notify-finish

Start web console

git-annex webapp --config web.browser=google-chrome-stable \
--notify-start --notify-finish

Move unused files to trash remote

git annex unused
git annex move --unused --to trash

List all local configuration options

git config --global --get-regexp '.*\..*'
git config --local --get-regexp '.*\..*'