[Libosinfo] Restructuring loading of the database

[Zeeshan Ali (Khattak)]

Hi Daniel,

On Thu, Sep 17, 2015 at 5:38 PM, Daniel P. Berrange <berrange at redhat.com> wrote:
> On Thu, Sep 17, 2015 at 02:06:31PM +0100, Daniel P. Berrange wrote:
>> As mentioned previously I'd like us to move towards splitting the database
>> off from the code
>>
>>   https://www.redhat.com/archives/libosinfo/2015-July/msg00010.html
>>
>> There were many items in that mail, but the one I want to address first
>> is the database file/dir structure and the way we load it.
>>
>> Currently we have 3 locations we'll load from by default:
>>
>>  - System path (aka $INSTALL_PREFIX/share/libosinfo/db)
>>  - Local path (aka $INSTALL_PREFIX/etc/libosinfo/db)
>>  - User path  (aka $XDG_CONFIG_DIR/libosinfo/db - usually $HOME/.config)
>>
>> We we process each directory in that order, and load all XML files we
>> find there. We have subdirectories for different types of entity, but
>> this is merely by convention, there is nothing in the loader code that
>> requires these named subdirectories.
>>
>> If there are XML files that use the same entity ID in multiple directories
>> we are entirely additive.
>>
>> eg if there is some extra info you want to record against an operating
>> system, you can put it in $HOME/.config/libosinfo/db and it will extend
>> the information already defined in /usr/share/libosinfo/db for that
>> operating system.
>>
>> libosinfo attributes are all multi-valued, so if you define a new value
>> for the an existing attribute, it is added to what's already there. There
>> is no mechanism to either remove data already defined, or to replace data.
>>
>> When thinking about alternative ideas, I realized that systemd has already
>> figured out a nice solution to this probem. Given a file "foo.unit" in
>> /lib/systemd/system/, if you create a file 'foo.unit' in /etc/systemd/system
>> it will completely replace the earlier defined unit. If you create a file
>> 'foo-custom.conf' in /etc/system/system/foo.unit.d, it will augment the
>> earlier defined unit. If you create a symlink /etc/systemd/system/foo.unit
>> pointing to /dev/null, it will completely delete the previously defined
>> unit.
>>
>> This works nicely, assuming you have one entity being defined per file.
>> So I'm suggesting we rework libosinfo to follow a similar scheme.
>>
>>  - One file per entity being defined.
>>  - The filename for an entity becomes part of the ABI of the database.
>>  - To completely replace an entity create a new file with the smae name
>>  - To augment the existing entity, create a file in a $FILE.d directory
>>  - To delete a previously entity, create a file symlinked to /dev/null.
>>
>> So, eg considering "fedora-20" entity:
>>
>>  - Master file /usr/share/libosinfo/db/os/fedora-20.xml
>>  - Replace it with /etc/libosinfo/db/os/fedora-20.xml
>>  - Augment it with /etc/libosinfo/db/os/fedora-20.d/foo.xml
>>  - Delete it with  /etc/libosinfo/db/os/fedora-20.d/foo.xml -> /dev/null
>>
>> I think we can do this without creating serious backcompat problems
>> with existing loader code, as we're not changing the XML schema. Existing
>> code will treat all files as augmenting existing info, so won't handle
>> the replace/delete semantics, but that's no worse than what they have
>> today.
>>
>> The only real question I have is whether we should define a formal
>> naming convention for the files. We could define a mapping based on
>> the ID URI for an entity. eg
>>
>>   <os id="http://fedoraproject.org/fedora/20">
>>
>> Goes into a file
>>
>>    fedoraproject.org-fedora-20.xml
>>
>> ie replace all non-a-Z0-9 characters in the URI with a -. This works
>> but I feel it isn't too user friendly, because the filenames get
>> pretty long. We could use the short-id defined for operating systems,
>> but other entity types don't have short-ids, and also there can be
>> multiple short-ids.
>>
>> So at this point my inclination, is to not have a formal requirement
>> for the file name. Simply a suggestion that the filename be like the
>> short-id or equivalent. The only requirement would be that we must
>> *NEVER* change the filename for a given entity once it is defined.
>> This ensures local user overrides for an entity, don't break when
>> the master database is updated.
>
> An alternative way would be to allow 'short-id' against any type of
> entity, not just OsinfoProduct subclasses, and then say that the
> first short-id listed in the file must be used as the name. This is
> somewhat appealing to me, as it gives clear guidance on naming.

Yeah, this sounds better to me too.

-- 
Regards,

Zeeshan Ali (Khattak)
________________________________________
Befriend GNOME: http://www.gnome.org/friends/