[Libosinfo] [RFC] Providing apps an easier way to update osinfo-db

[Fabiano Fidêncio]

On Tue, Jul 3, 2018 at 1:54 PM, Daniel P. Berrangé <berrange at redhat.com> wrote:
> On Tue, Jul 03, 2018 at 01:32:28PM +0200, Fabiano Fidêncio wrote:
>> Folks,
>>
>> One thing that has been discussed between Felipe Borges (GNOME Boxes
>> maintainer) and myself is the possibility to, from an app, download
>> and install the latest osinfo-db content without being dependent of
>> distribution packagers and being able to ensure that the users will
>> have the most up-to-date db as soon as a something was
>> introduced/changed.
>>
>> In order to do so, we'd have to (according to my understanding):
>> - Support URLs as arguments for osinfo-db-import:
>>   This step would allow the apps to download the latest release from
>> our official source;
>
> This is fine, and needed no matter what else is done I think.
>
>> - Expose a "latest" osinfo-db build:
>>   Currently we tag our releases, make a tarball and this tarball is
>> uploaded in pagure. The tarball's name looks like
>> "osinfo-db-20180612.tar.xz" and, preferably, we'd like to have
>> something like "osinfo-db-latest.tar.xz" pointing to the latest
>> release. Actually, in an ideal world, we'd like to have a
>> osinfo-db-latest.tar.xz pointing to the last osinfo-db commit (then,
>> maybe, we would have to rely on gitlab infra to do that?);
>
> I think the idea of a -latest.tar.xz link is a mistake as it
> encourages apps to download the content even if hasn't been changed,
> as they can't easily see if it is a new version or not.

With the -latest naming I was envisioning that we could ship
non-official releases, but git master.
So, first thing, the name is, at least, misleading. :-/

Daniel,
You think that shipping git master would also be a mistake? We can
easily do a release per day we have patches and update the release in
pagure without issues. But it becomes problematic in order to how
distribute those to fedora (considering that the time waiting for
those to reach the distro is ~2 weeks from the release day ... and I'm
packaging it as soon as a release is done).

If you think this idea is okay, how do you envision it being
shipped/displayed to the apps to download? The reason I'm asking here
is because if we take this path, we'd have to have a way to provide
git master already built to the apps and having this on pagure, for
sure, doesn't seem like a good idea.

>
>> - Provide a DBus API for osinfo-db-import and osinfo-db-validate (at least):
>>   With this we can easily spawn, for instance, `osinfo-db-import
>> https://releases.pagure.org/libosinfo/osinfo-db-latest.tar.xz` from
>> any app using osinfo-db;
>
> I'm pretty sceptical about need for adding dbus into this. I don't see
> why an app couldn't just spawn the commands as is. Turning them into
> a DBus service feels like massive overkill for the problem.

Okay from my side.
I'd like to hear here from Felipe as well why a dbus API would be
their preferred way and, mainly, whether we could go for spawning the
command as they are now.

>
> Also we must not have any applications hardcoding releases.pagure.org
> URLs. I do not have confidence in pagure.org sticking around over the
> long time frames RHEL lives for.
>
>> Ideas? Possible limitations?
>
> I've been thinking about live updates for quite a while - ever since I
> created the osinfo-db-tools separate package in fact :-)
>
> The two key problems with any live update mechanism are scalability and
> compatibility. We need to ensure that applications don't repeatedly
> hit the web server to download unchanged content over & over. We also
> neeed to ensure that we don't bake in usage of a URL that is liable
> to change over time. Whatever we do will end up in RHEL releases that
> can live for 10+ years, and I don't have confidence our hosting will
> live for 10+ years without moving at least once.
>
> My thought was to leverage DNS as a key part of the solution.  Have DNS
> TXT record exposing the current live update URL ie a TXT query against
> "dbupdate.libosinfo.org"  could return something akin to
>
>   url=https://releases.pagure.org/libosinfo/osinfo-db-20180612.tar.xz,version=20180612
>
> The locally install osinfo-db content always creates a file VERSION in
> the root of the install location. So that content can be compared against
> the DNS result to see if it is outdated of not, and thus avoid hitting
> the web server at all unless there's genuinely a change that needs
> acquiring.
>
> I would anticipate some osinfo-db-update-check tool to perform this work
> and print out a URL, that could then be given to osinfo-db-import.

I like the idea!

Who's the owner of libosinfo.org? Daniel? Red Hat?

Daniel,
Did you start poking at this in the past? If not, it's something I can
slowly starting to take a look at.

>
> It feels like a cron job to kick off periodic updates would be nice, but
> the downside with that is that it synchronizes a DOS attack from all
> deployed hosts (modulo timezone differences).

The idea about what will be used on Boxes is something to be discussed
during its hackfest that will  happen next Monday.
I'm expecting that for generating the flatpack a cron job will be
used. I'm not sure whether we want to also have an option to update
the osinfo-db on non-flatpak'ed apps (aka, the ones normally shipped
on OSes), although I do believe GNOME Boxes should!

>
> Security is, however, a key issue here. For this to be practical at the
> very least it would likely require either DNSSEC, or embedding a crypto
> signature in the result. The latter would however require a long term
> pub key to be embedded in the client, and I'm not a fan of that.
>
> There is a concern of reinventing the wheel here. There is some kind of
> industry standard I vaguely recall for doing software/content updates
> over the internet. I've not investigated that in enough detail to
> understand if it avoids the scalability & compatibility concerns I
> have though

Hmm. Right, it makes sense. I'll check what the standard is and how we
could take advantage of that.
.
>
> Regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|


Best Regards,
-- 
Fabiano Fidêncio