[Libosinfo] Restructuring loading of the database

[Daniel P. Berrange]

As mentioned previously I'd like us to move towards splitting the database
off from the code

  https://www.redhat.com/archives/libosinfo/2015-July/msg00010.html

There were many items in that mail, but the one I want to address first
is the database file/dir structure and the way we load it.

Currently we have 3 locations we'll load from by default:

 - System path (aka $INSTALL_PREFIX/share/libosinfo/db)
 - Local path (aka $INSTALL_PREFIX/etc/libosinfo/db)
 - User path  (aka $XDG_CONFIG_DIR/libosinfo/db - usually $HOME/.config)

We we process each directory in that order, and load all XML files we
find there. We have subdirectories for different types of entity, but
this is merely by convention, there is nothing in the loader code that
requires these named subdirectories.

If there are XML files that use the same entity ID in multiple directories
we are entirely additive.

eg if there is some extra info you want to record against an operating
system, you can put it in $HOME/.config/libosinfo/db and it will extend
the information already defined in /usr/share/libosinfo/db for that
operating system.

libosinfo attributes are all multi-valued, so if you define a new value
for the an existing attribute, it is added to what's already there. There
is no mechanism to either remove data already defined, or to replace data.

When thinking about alternative ideas, I realized that systemd has already
figured out a nice solution to this probem. Given a file "foo.unit" in
/lib/systemd/system/, if you create a file 'foo.unit' in /etc/systemd/system
it will completely replace the earlier defined unit. If you create a file
'foo-custom.conf' in /etc/system/system/foo.unit.d, it will augment the
earlier defined unit. If you create a symlink /etc/systemd/system/foo.unit
pointing to /dev/null, it will completely delete the previously defined
unit.

This works nicely, assuming you have one entity being defined per file.
So I'm suggesting we rework libosinfo to follow a similar scheme.

 - One file per entity being defined.
 - The filename for an entity becomes part of the ABI of the database.
 - To completely replace an entity create a new file with the smae name
 - To augment the existing entity, create a file in a $FILE.d directory
 - To delete a previously entity, create a file symlinked to /dev/null.

So, eg considering "fedora-20" entity:

 - Master file /usr/share/libosinfo/db/os/fedora-20.xml
 - Replace it with /etc/libosinfo/db/os/fedora-20.xml
 - Augment it with /etc/libosinfo/db/os/fedora-20.d/foo.xml
 - Delete it with  /etc/libosinfo/db/os/fedora-20.d/foo.xml -> /dev/null

I think we can do this without creating serious backcompat problems
with existing loader code, as we're not changing the XML schema. Existing
code will treat all files as augmenting existing info, so won't handle
the replace/delete semantics, but that's no worse than what they have
today.

The only real question I have is whether we should define a formal
naming convention for the files. We could define a mapping based on
the ID URI for an entity. eg

  <os id="http://fedoraproject.org/fedora/20">

Goes into a file

   fedoraproject.org-fedora-20.xml

ie replace all non-a-Z0-9 characters in the URI with a -. This works
but I feel it isn't too user friendly, because the filenames get
pretty long. We could use the short-id defined for operating systems,
but other entity types don't have short-ids, and also there can be
multiple short-ids.

So at this point my inclination, is to not have a formal requirement
for the file name. Simply a suggestion that the filename be like the
short-id or equivalent. The only requirement would be that we must
*NEVER* change the filename for a given entity once it is defined.
This ensures local user overrides for an entity, don't break when
the master database is updated.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|