That's why it's there, after all. This also makes the system
more robust and catches some more errors.
The next thing this enables is to also allow right-clicking
on refs in the commit list.
The ref functionality is somewhat weird right now, in that
some parts of the code require a nil array when there are no
refs. So, we add a good removeRef accessor to keep track of this