“Git Cat” Cont’d

A comment from HappyGiraffe on the previous post pointed out that the “show” verb would accept the commit hash along with the filename to return the same result I was looking. I had actually tried this out originally but ran into the problem where I couldn’t get the syntax correct. After re-reading the man page for the show verb, it turns out I needed to specify the full path for the filename relative to the top of the repository tree:

git show commit_id:some/path/to/file

The problem becomes getting the full path to the file in the repository. Here we really get to see my inexperience with Git and HappyGiraffe came to the rescue again. He pointed out that the “ls-file” verb should do the trick. After a glance at the man page, it turns out the syntax to get the full path for a file should look like this:

git ls-file –full-name file

So now rather than digging out the blob hash id from the commit, we can just specify the commit id and the full path of the file. To get the commit id from git, it’s a matter of dumping the log and doing some awk work.
But I got curious about whether it was possible to dump just the commit has for a log entry and it turns out that you can using the following syntax:

git log -1 –pretty=format:%H file

The “-1” argument only displays the last log entry for that particular file. The “pretty” option specifies that we’re going to format the output in a paticular way and the “%H” argument to it returns the full hash (a lower case “h” only returns a partial on the hash id). Now we don’t have to pipe it to awk for any string hackery.

So the end result becomes (in pseudo shell script):

commit_id = git log -1 –pretty=format:%H file
fullpath = git ls-file –full-name file
git show commit_id:fullpath

Which is excellent. Though… it’d still be nice if the folks maintaining git would provide us with a “cat” verb.

No “svn cat” in git.

A couple of years ago, I was working with a guy who write a little script that I’ve found completely invaluable. It simply did the following:

• Accepted a filename of a file in a svn working copy tree.
• Grabbed the previous checked in version of the file.
• Opened the two files in the Filemerge utility on a mac.

Filemerge is a great utility for looking at diffs side by side. From the command line, you use the opendiff command to get at it via scripting.

However, with all the recent work I’ve been doing with Git I’ve been been without my nice little tool. Yesterday I finished up some work on parsing xml and found that the diff via the terminal was just a little too much to go through. So I decided to hack up a Git version of the tool and add a couple of enhancements. Turns out not so easy.

The problem was that Git doesn’t have a simple way to output the checked in copy of a particular file. The previous version of the script used “svn cat” to output from revision control. Git does not have an equivalent. So I hacked my own version using three different Git commands in the process.

To get the equivalent of “svn cat” in git, you need to do the following commands:

1. svn log
   This will print out a list of the commits that the file has been associated with. Your most concerned with the first commit listed in the log (which represents the last commit). Grab the hash for that commit. Should look like this:

   commit 9978b72045c139b80eab960f7381502fa2106374

2. svn ls-tree
   This will list the files associated with that commit and give you the hashes for the particular version of that file. Grab the hash for the particular file you’re interested in. Should look like this:

   100644 blob 342c0eacda335f17d21275f657e827c83e5e3f48 filename

3. svn show -p
   This will print to stdout the contents of that file at that version. This is what we’ve been looking for.

Viola. Now I have the contents of the previous checked in version of the file and am happy as a clam. I was able to get the script working after that.

When I can get around to spending a little time cleaning up the code, I’ll publish it here for folks to use.

More Git Public Repo Stuff

I’m really enjoying GIT. It’s makes creating and using private local repo easy while still enforcing some best practices managing code. However, I ran into a bit of clunky workflow when creating a public repo out of a private local one.

I’m managing some code in a local repo on my laptop. Having gotten the code to the point where I want to do some continuous integration, I decided to follow the instructions for setting up a public repository from an existing repository. Pretty simple stuff to do the “bare” clone of the repo and push it to the public server where it’ll be hosted. However, to push new changes to code I would need to type something long winded like this:

$ git push ssh://yourserver.com/~you/proj.git master

or if I setup the following in my local repository config:

$ cat >>.git/config <<EOF [remote “public-repo”] url = ssh://yourserver.com/~you/proj.git EOF

It would be as simplify it bit as:

$ git push public-repo master

Which is pretty straight forward but not very simple. A simple “git push” would be what I’m aiming for. So at this point, I decided to see what else I could do to get from the verbose push of the last example to just “git push”. Turns out it’s pretty simple with a couple of extra steps of effort.

So with the public repository available remotely, clone a fresh copy of the repo into temporary directory. From there, look in the .git/config file and copy the following from that file:

[remote “origin”]
url = ssh://user@yourserver.com/~you/proj.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch “master”]
remote = origin merge = refs/heads/master

and past that into your original .git/config file in the repo you cloned for the public repository.

Once you do that, you only simply need to do a “git push” in order to push your local changes to your public repository. You no longer have to specify the branch or the “project” name. Which makes it Simple.

Continious Integration with Buildbot and Git

My test setup is evolving. I now need to play with buildbot and git integration to get some level of automated build and test management. The only examples of git integration I could find with buildbot and git were the meta buildbot instance and this blog post. The prior’s config has not been published yet (though it appears to be grabbing changes via 4 hour scheduler) and the later is a bit confusing. So I thought I’d document my walk through this little situation since I needed to iron out how it all would work.

Side note: One of continuing issues I have with using buildbot is that while the documentation is great, there are not enough examples of usage for the various version control systems. Having had to set it up for perforce, subversion and now git, it’d be nice to see what folks had previously done. Me thinks we need a cook book for that.

Now this is a test setup to iron out the kinks of a basic work flow. These instructions are a guide for getting started and the setup would need a lot more tuning for a production environment. This post assumes the following:

• that you have an instance of the buildbot server up and running.
• That you have atleast one buildbot slave attached to the buildbot server.
• that you have public git repo you administrate.
• That you have ssh keys setup so that you don’t have to enter passwords when you do a pull to your slave machine from the git repo.

If you don’t have a buildbot instance, go set one up using these instructions. If you don’t have a test setup for a public git repo, take a look at this previous entry in this blog (which also has info on setting up ssh keys).

Here we go:

1. In the buildbot/contrib directory, you will find git_buildbot.py. Copy this to the remote server your public git repo is being hosted on. You need to make sure that python and twisted are installed on this host, since the script uses the twisted frame for network interactions. Make sure to edit master variable in the script to point to the server hostname and listening port of your buildbot instance. Example:

master="some.remotehost.com:9989"

2. On the remote server, create a post-receive hook script file in your public repo and make sure it’s executable. You would do this under .git/hooks and point the script to run the git_buildbot.py script with arguments. Some like this:

/path/to/script/git_buildbot.py $1 $2 $3

3. Once that’s done, you’ll need to edit your buildbot instance’s master.cfg file to add a PBChangesource statement. This will have buildbot listen for the changes being sent by the git_buildbot.py script. Do this:

from buildbot.changes.pb import PBChangeSource
c['change_source'] = PBChangeSource()

4. Next we’ll add a scheduler to catch the changes. Make sure that you specify the branch as master, otherwise your changes will get ignored by the scheduler:

c['schedulers'] = []
test_schedule = Scheduler(name="test", branch="master", treeStableTimer=0, builderNames=["test-builder"])
c['schedulers'].append(test_schedule)

5. Now we’ll add a build factory that will get executed when the changes occur. For simplicity sake, we’ll just pull (or fetch in the case of buildbot) the latest revision of the code. Note that the repourl we’re using is prefaced by ssh. This is why you’ve previously setup your ssh keys between your slave and the git repo:

test_factory = factory.BuildFactory()
test_factory.addStep(Git(repourl="ssh://gitrepo.somehost.com/home/jedi/gitrepo"))

6. Restart your buildbot server and make sure your slave is attached.
7. Checkout, edit, commit and push a file in the code base your using to test this setup. You should see the following:

• Buildbot gets the change notice from the post-receive hook.
• Buildbot hands the change off to the builder.
• The buildslave runs the job and successfully gets the code update.

And that’s it. Just a reminder, make sure your hook set to be executable on the git public repo. This tripped me up for a couple of minutes until I reread the docs, so it’s probably a good idea to do a bit more reading on hooks in git. They’re pretty much the same as subversion, only just more of them.

Quickly setting up a Public Git Repository

Recently, I’ve needed to learn how to administrate a Git repo for a contracting gig. Since I’ve had zero experience with git beyond submitting a documentation patch for buildbot, I decided to put together a test setup. The goal was to have a test public repo on a remote server and pull a copy of that repo it to my local dev machine. I was actually surprised at how quick and easy this was to do.

The following assumes that you are in unix-like environments (linux, osx, cygwin), that you have a remote server and a local dev machine, and that you’re going to use ssh for your access to the remote server.

1. Install Git on both the remote server and your local development box.
2. On the remote server, we’re going to setup a public repo. Now if you’re planing to do an import of an existing code base with history, that’s kind of beyond the scope of this doc. If you’ve got some code to just import, clean up any scm metadata files.
3. In the top level of that code base, do the following:


RemoteServer:~> cd gitrepo/
RemoteServer:~/gitrepo> git init
Initialized empty Git repository in /home/jedi/gitrepo/.git/
RemoteServer:~/gitrepo> git add .
RemoteServer:~/gitrepo> git commit

4. When you commit, enter your commit message in your favorite editor and let it do it’s work.
5. Next, on your development box do the following:


[devhost:~] jedi% git clone ssh://some.remoteserver.com/home/jedi/gitrepo
gitrepo
Initialized empty Git repository in /Users/jedi/gitrepo/.git/
remote: Counting objects: 410, done.
remote: Compressing objects: 100% (356/356), done.
remote: Total 410 (delta 46), reused 0 (delta 0)
Receiving objects: 100% (410/410), 457.06 KiB | 249 KiB/s, done.
Resolving deltas: 100% (46/46), done.
[devhost:~] jedi%


You should now have the code in your local directory and can now play with running & managing a public git repo.

However, the setup is quite simple and doesn’t solve all problems when you need to setup a production ready public repo. For further information about getting to production ready, check out the official docs regarding the setup of a public repository. Also helpful will be documentation on setting up ssh keys so that you’re not entering your password each time you do any operation with the remote repository.

svn+ssh on windows

Without TortoiseSVN. Yep. Straight up command line.

This is a problem that’s plagued me for a while. I’ve tried to work out the kinks before, only to be frustrated by the various ways that you can affect the configuration on the windows host. I believe I’ve finally got the solution and I’m interested to find how many other folks run into this problem and have a different solution.

So without further ado, try the following:

1. Download the windows version of the command line client.
2. Download the putty suite of tools.
3. Generate a new key pair on the remote unix server acting as the repo.
4. Copy the private key from the remote server and convert it using puttygen.
5. In your svn config file, include the following in the tunnel section (or where ever you’ve placed the key):

ssh=”C:/Program Files/PuTTY/plink.exe” -i “C:/Program Files/PuTTY/id_dsa.ppk”

6. Now use the svn+ssh syntax and list a url in the repo.

You’re good to go.

For more information on generating ssh keys, check out this documentation.

UPDATE

Funny. Just posted this and pointed it out to a friend. He immediately responds with the following link:

http://agateau.wordpress.com/2007/07/03/windows-svnssh-and-the-subversion-command-line-client/

Same steps, lot more detail. I defer to the better example in this case. 😉