[Libosinfo] Restructuring loading of the database

[Daniel P. Berrange]

On Thu, Sep 17, 2015 at 02:06:31PM +0100, Daniel P. Berrange wrote:
> As mentioned previously I'd like us to move towards splitting the database
> off from the code
> 
>   https://www.redhat.com/archives/libosinfo/2015-July/msg00010.html
> 
> There were many items in that mail, but the one I want to address first
> is the database file/dir structure and the way we load it.
> 
> Currently we have 3 locations we'll load from by default:
> 
>  - System path (aka $INSTALL_PREFIX/share/libosinfo/db)
>  - Local path (aka $INSTALL_PREFIX/etc/libosinfo/db)
>  - User path  (aka $XDG_CONFIG_DIR/libosinfo/db - usually $HOME/.config)
> 
> We we process each directory in that order, and load all XML files we
> find there. We have subdirectories for different types of entity, but
> this is merely by convention, there is nothing in the loader code that
> requires these named subdirectories.
> 
> If there are XML files that use the same entity ID in multiple directories
> we are entirely additive.
> 
> eg if there is some extra info you want to record against an operating
> system, you can put it in $HOME/.config/libosinfo/db and it will extend
> the information already defined in /usr/share/libosinfo/db for that
> operating system.
> 
> libosinfo attributes are all multi-valued, so if you define a new value
> for the an existing attribute, it is added to what's already there. There
> is no mechanism to either remove data already defined, or to replace data.
> 
> When thinking about alternative ideas, I realized that systemd has already
> figured out a nice solution to this probem. Given a file "foo.unit" in
> /lib/systemd/system/, if you create a file 'foo.unit' in /etc/systemd/system
> it will completely replace the earlier defined unit. If you create a file
> 'foo-custom.conf' in /etc/system/system/foo.unit.d, it will augment the
> earlier defined unit. If you create a symlink /etc/systemd/system/foo.unit
> pointing to /dev/null, it will completely delete the previously defined
> unit.
> 
> This works nicely, assuming you have one entity being defined per file.
> So I'm suggesting we rework libosinfo to follow a similar scheme.
> 
>  - One file per entity being defined.
>  - The filename for an entity becomes part of the ABI of the database.
>  - To completely replace an entity create a new file with the smae name
>  - To augment the existing entity, create a file in a $FILE.d directory
>  - To delete a previously entity, create a file symlinked to /dev/null.
> 
> So, eg considering "fedora-20" entity:
> 
>  - Master file /usr/share/libosinfo/db/os/fedora-20.xml
>  - Replace it with /etc/libosinfo/db/os/fedora-20.xml
>  - Augment it with /etc/libosinfo/db/os/fedora-20.d/foo.xml
>  - Delete it with  /etc/libosinfo/db/os/fedora-20.d/foo.xml -> /dev/null
> 
> I think we can do this without creating serious backcompat problems
> with existing loader code, as we're not changing the XML schema. Existing
> code will treat all files as augmenting existing info, so won't handle
> the replace/delete semantics, but that's no worse than what they have
> today.
> 
> The only real question I have is whether we should define a formal
> naming convention for the files. We could define a mapping based on
> the ID URI for an entity. eg
> 
>   <os id="http://fedoraproject.org/fedora/20">
> 
> Goes into a file
> 
>    fedoraproject.org-fedora-20.xml
> 
> ie replace all non-a-Z0-9 characters in the URI with a -. This works
> but I feel it isn't too user friendly, because the filenames get
> pretty long. We could use the short-id defined for operating systems,
> but other entity types don't have short-ids, and also there can be
> multiple short-ids.
> 
> So at this point my inclination, is to not have a formal requirement
> for the file name. Simply a suggestion that the filename be like the
> short-id or equivalent. The only requirement would be that we must
> *NEVER* change the filename for a given entity once it is defined.
> This ensures local user overrides for an entity, don't break when
> the master database is updated.

An alternative way would be to allow 'short-id' against any type of
entity, not just OsinfoProduct subclasses, and then say that the
first short-id listed in the file must be used as the name. This is
somewhat appealing to me, as it gives clear guidance on naming.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|