Keybase Git

(For a gentler introduction to Keybase Git, see our launch announcement.)

Overview

Keybase can store end-to-end encrypted git repositories, in a way that's compatible with the standard git suite of commands. New git repos can be created in the app under your private namespace (in which case they are encrypted for only your devices), or in one your teams. Team repos can be read by all members of that team, and can be pushed to by all members with a role of writer or above. You can create a repo in the Git tab of the app, or from a terminal with keybase git commands, from any device where you have Keybase installed and you are logged in. For example, to create a private repo called dotfiles:

$ keybase git create dotfiles
Repo created! You can clone it with:
  git clone keybase://private/you/dotfiles
Or add it as a remote to an existing repo with:
  git remote add origin keybase://private/you/dotfiles

You can use the resulting repo URL (in this case, "keybase://private/you/dotfiles") as a remote URL, in the same way you would use any "https" or "git@" remote, from any device where you have Keybase installed and you are logged in.

The names of your git repositories are also encrypted just for you or your team -- the Keybase servers and admins cannot see them.

Implementation

Under the covers, Keybase Git is very simple. It stores a git bare repo (just like the one you would make with git init --bare) in the corresponding KBFS folder for the namespace you chose (either your private KBFS folder, or the team's KBFS folder). It simply creates a hidden subdirectory in that folder (.kbfs_git), and stores each repository in there as a subdirectory. KBFS handles all of the encryption and decryption for us. The only difference is that we tag the resulting storage as git-specific, so that it counts against a different quota than your normal KBFS data. Each user and each team gets 100 GB of git data.

To funnel commands from normal git commands to Keybase, we implemented a git remote helper, in the form of an executable called git-remote-keybase that ships with Keybase. git-remote-keybase is invoked whenever you do a git operation with a "keybase://" remote URL. It uses the amazing go-git package to manipulate the bare git repos stored on KBFS.

The source code is available here and here. Because of the unique way we want to report upload and download progress on the command line, we maintain a fork of go-git.

Browsing your repos directly inside KBFS (a.k.a. Autogit)

With Keybase Git, you can browse the data in any of your repositories, without actually needing to create a clone of that repository. Not only does this let you browse the repo in the Keybase app on your computer or phone, but it also lets you access the files on the command line. We call this feature Autogit, and there is a special .kbfs_autogit directory available in the KBFS folder that magically gives you read-only access to the repositories. For example:

$ ls -l /keybase/private/you/.kbfs_autogit/dotfiles/.bashrc
-r-------- 1 you root 3875 Nov 22 10:26 /keybase/private/you/.kbfs_autogit/dotfiles/.bashrc
$ head /keybase/private/you/.kbfs_autogit/dotfiles/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

It's also a nice way to expose checked-in git files directly in KBFS, via symlink:

$ cd /keybase/private/you
$ ln -s .kbfs_autogit/dotfiles/.bashrc my-bashrc
$ head my-bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

By default autogit browses the master branch of your respository, though you can browse a particular branch by using a .kbfs_autogit_branch_[branchname] as a subdirectory, like this (for a branch named sl):

$ head /keybase/private/you/.kbfs_autogit/dotfiles/.kbfs_autogit_branch_sl/.bashrc
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples

alias ls=sl

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;

You can also see the commit message and diff for a give commit hash, like this:

$ cat /keybase/private/you/.kbfs_autogit/dotfiles/.kbfs_autogit_commit_524e1315cc8e01c6db5b5fcd66475614356523d8
commit 524e1315cc8e01c6db5b5fcd66475614356523d8
Author: Your Name <you@you.com>
Date:   Fri Nov 22 10:42:39 2019 -0800

    sl

diff --git a/.bashrc b/.bashrc
index 83d427aa03335489a58ff0a007f3972c99175d27..fb32d2acd02399091de7d32c692fa53c28091423 100644
--- a/.bashrc
+++ b/.bashrc
@@ -2,6 +2,8 @@ # ~/.bashrc: executed by bash(1) for non-login shells.
 # see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
 # for examples

+alias ls=sl
+
 # If not running interactively, don't do anything
 case $- in
     *i*) ;;

You need to provide the full commit hash in the path; the short hash won't work. Also, due to technical limitations, we cannot show the very first commit of the repository this way.

Support for Git LFS (Large File Storage)

The Git LFS project is a git extension that allows you to store pointers to large files in a git repo, to keep repository sizes and clone times more manageable.

As of Keybase 5.1.0, you can configure LFS to store those large files in KBFS. All you need to do is to install and configure LFS, have a local git repo with a "keybase://" remote URL, and run a quick configuration command. Here's an example (the first four steps are standard LFS configuration steps, and are not specific to Keybase):

  1. Download and install LFS.
  2. Run git lfs install in your repository.
  3. Mark which file types should be tracked by LFS, e.g.: git lfs track "*.psd".
  4. Add the attributes file: git add .gitattributes.
  5. Within the repository (assuming it already has a Keybase remote URL): keybase git lfs-config.

And that's it! Your normal git flow should push and pull the LFS-tracked files from your Keybase git repo with no more effort.

Note that anyone who clones the repo will get an error message about an unknown LFS protocol. This is unavoidable given that Keybase is adding a custom protocol that is not natively supported by LFS. But after you clone, all you need to do run keybase git lfs-config in the local repo, and do another checkout, like this:

$ git clone keybase://private/you/project
Cloning into 'project'...
Initializing Keybase... done.
Syncing with Keybase... done.
Syncing encrypted data to Keybase: (100.00%) 139.33/139.33 KB... done.
Counting: 74.07 KB... done.
Cryptographic cloning: (100.00%) 74.07/74.07 KB... done.
Downloading bigfile.psd (90 MB)
Error downloading object: bigfile.psd (cbf1cad): Smudge error: Error downloading bigfile.psd (cbf1cad3c09c22c714222c474a5cdc8a44e0fe5a7ff2874a1d50a74e95d9c5bd): batch request: missing protocol: "keybase://private/you/project.git/info/lfs"

Errors logged to project/.git/lfs/logs/20191122T112245.269637036.log
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: bigfile.psd: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

$ cd project
$ keybase git lfs-config
Success! This repo is now configured to use the following Keybase
repository for LFS:
    keybase://private/you/project

Assuming you have installed Git LFS (see https://git-lfs.github.com) you can now
configure git to store certain files directly in Keybase.  For example:
    git lfs install
    git lfs track "*.zip"
    git add .gitattributes

Note that new checkouts of this repository will see a "missing protocol" error
until you have configured it with this `keybase git lfs-config` command.  After
doing so, you can sync the LFS files with this command:
    git checkout -f HEAD

$ git checkout -f HEAD
Filtering content: 100% (13/13), 855.76 MiB | 22.61 MiB/s, done.
Your branch is up to date with 'origin/master'.

Note that you can supply a custom remote to keybase git lfs-config with the --repo flag if the default one isn't what you want, and you can also do an out-of-place config with the --path flag.

You can use Keybase Git LFS for large-file storage even for repositories that are hosted on other, non-Keybase services too, if you want. However, you still must create a git repo in Keybase and add that remote to your local repository.

LFS files count toward your KBFS git quota.

Public and shared private repos

The Keybase app and command line support creating private repos, and team repos. If you want to make a public repo, or a private repo shared with other individuals, you can do that, but it's not officially supported. You won't be able to create, list or delete those repos using the Keybase app. But if you really want one, you can just push to the appropriate URL (e.g., "keybase://public/you/project" or "keybase://private/you,them/project"), and it should just work.

Chat notifications

You can get Keybase chat notifications when someone pushes to your Keybase Git team repo. These notifications include the name, date, commit message header and branch of the commit, along with a link to browse the commit itself. By default these go into the #general channel of the team chat, though admins can change that to a different channel (or disable that entirely) in the Git tab of the Keybase app.

Debugging

If you run into a problem with Keybase git, your best course of action is to send feedback with a description of the problem, and logs, to Keybase admins in the app or with keybase log send.

As mentioned above, Keybase stores the underlying bare repo in KBFS. If you are having issues and want to look at the bare repo directly, you can just cd into the .kbfs_git subdirectory of the corresponding private or team folder where the repo lives. But reader beware: this gives full read-write access directly to the bare repo, so if you delete or alter anything in this directory, it will alter the git repo itself. However, you might find it useful for debugging; just be very careful in there.

When you delete a repository from Keybase (via the app or keybase git delete), it just gets moved out of the way to a hidden directory called .kbfs_git/.kbfs_deleted_repos, where it is cleaned up lazily in the background after a short amount of time has passed. If you're having trouble with a deleted repo, or seeing a message about a repo that can't be deleted, you can look in there and either delete it manually, or rename it to a different name so that it will be ignored by the background process.