[Libosinfo] [RFC] Providing apps an easier way to update osinfo-db

[Daniel P. Berrangé]

On Tue, Jul 03, 2018 at 01:32:28PM +0200, Fabiano Fidêncio wrote:
> Folks,
> 
> One thing that has been discussed between Felipe Borges (GNOME Boxes
> maintainer) and myself is the possibility to, from an app, download
> and install the latest osinfo-db content without being dependent of
> distribution packagers and being able to ensure that the users will
> have the most up-to-date db as soon as a something was
> introduced/changed.
> 
> In order to do so, we'd have to (according to my understanding):
> - Support URLs as arguments for osinfo-db-import:
>   This step would allow the apps to download the latest release from
> our official source;

This is fine, and needed no matter what else is done I think.

> - Expose a "latest" osinfo-db build:
>   Currently we tag our releases, make a tarball and this tarball is
> uploaded in pagure. The tarball's name looks like
> "osinfo-db-20180612.tar.xz" and, preferably, we'd like to have
> something like "osinfo-db-latest.tar.xz" pointing to the latest
> release. Actually, in an ideal world, we'd like to have a
> osinfo-db-latest.tar.xz pointing to the last osinfo-db commit (then,
> maybe, we would have to rely on gitlab infra to do that?);

I think the idea of a -latest.tar.xz link is a mistake as it
encourages apps to download the content even if hasn't been changed,
as they can't easily see if it is a new version or not.

> - Provide a DBus API for osinfo-db-import and osinfo-db-validate (at least):
>   With this we can easily spawn, for instance, `osinfo-db-import
> https://releases.pagure.org/libosinfo/osinfo-db-latest.tar.xz` from
> any app using osinfo-db;

I'm pretty sceptical about need for adding dbus into this. I don't see
why an app couldn't just spawn the commands as is. Turning them into
a DBus service feels like massive overkill for the problem.

Also we must not have any applications hardcoding releases.pagure.org
URLs. I do not have confidence in pagure.org sticking around over the
long time frames RHEL lives for.

> Ideas? Possible limitations?

I've been thinking about live updates for quite a while - ever since I
created the osinfo-db-tools separate package in fact :-)

The two key problems with any live update mechanism are scalability and
compatibility. We need to ensure that applications don't repeatedly
hit the web server to download unchanged content over & over. We also
neeed to ensure that we don't bake in usage of a URL that is liable
to change over time. Whatever we do will end up in RHEL releases that
can live for 10+ years, and I don't have confidence our hosting will
live for 10+ years without moving at least once.

My thought was to leverage DNS as a key part of the solution.  Have DNS
TXT record exposing the current live update URL ie a TXT query against
"dbupdate.libosinfo.org"  could return something akin to

  url=https://releases.pagure.org/libosinfo/osinfo-db-20180612.tar.xz,version=20180612

The locally install osinfo-db content always creates a file VERSION in
the root of the install location. So that content can be compared against
the DNS result to see if it is outdated of not, and thus avoid hitting
the web server at all unless there's genuinely a change that needs
acquiring.

I would anticipate some osinfo-db-update-check tool to perform this work
and print out a URL, that could then be given to osinfo-db-import.

It feels like a cron job to kick off periodic updates would be nice, but
the downside with that is that it synchronizes a DOS attack from all
deployed hosts (modulo timezone differences).

Security is, however, a key issue here. For this to be practical at the
very least it would likely require either DNSSEC, or embedding a crypto
signature in the result. The latter would however require a long term
pub key to be embedded in the client, and I'm not a fan of that.

There is a concern of reinventing the wheel here. There is some kind of
industry standard I vaguely recall for doing software/content updates
over the internet. I've not investigated that in enough detail to
understand if it avoids the scalability & compatibility concerns I
have though.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|