[Libosinfo] [RFC] Providing apps an easier way to update osinfo-db

[Fabiano Fidêncio]

On Wed, Jul 4, 2018 at 8:46 AM, Fabiano Fidêncio <fabiano at fidencio.org> wrote:
> On Tue, Jul 3, 2018 at 1:54 PM, Daniel P. Berrangé <berrange at redhat.com> wrote:
>> On Tue, Jul 03, 2018 at 01:32:28PM +0200, Fabiano Fidêncio wrote:
>>> Folks,
>>>
>>> One thing that has been discussed between Felipe Borges (GNOME Boxes
>>> maintainer) and myself is the possibility to, from an app, download
>>> and install the latest osinfo-db content without being dependent of
>>> distribution packagers and being able to ensure that the users will
>>> have the most up-to-date db as soon as a something was
>>> introduced/changed.
>>>
>>> In order to do so, we'd have to (according to my understanding):
>>> - Support URLs as arguments for osinfo-db-import:
>>>   This step would allow the apps to download the latest release from
>>> our official source;
>>
>> This is fine, and needed no matter what else is done I think.
>>
>>> - Expose a "latest" osinfo-db build:
>>>   Currently we tag our releases, make a tarball and this tarball is
>>> uploaded in pagure. The tarball's name looks like
>>> "osinfo-db-20180612.tar.xz" and, preferably, we'd like to have
>>> something like "osinfo-db-latest.tar.xz" pointing to the latest
>>> release. Actually, in an ideal world, we'd like to have a
>>> osinfo-db-latest.tar.xz pointing to the last osinfo-db commit (then,
>>> maybe, we would have to rely on gitlab infra to do that?);
>>
>> I think the idea of a -latest.tar.xz link is a mistake as it
>> encourages apps to download the content even if hasn't been changed,
>> as they can't easily see if it is a new version or not.
>
> With the -latest naming I was envisioning that we could ship
> non-official releases, but git master.

Argh. "(...) non-official releases, like git master."
Sorry for the typo.

> So, first thing, the name is, at least, misleading. :-/
>
> Daniel,
> You think that shipping git master would also be a mistake? We can
> easily do a release per day we have patches and update the release in
> pagure without issues. But it becomes problematic in order to how
> distribute those to fedora (considering that the time waiting for
> those to reach the distro is ~2 weeks from the release day ... and I'm
> packaging it as soon as a release is done).
>
> If you think this idea is okay, how do you envision it being
> shipped/displayed to the apps to download? The reason I'm asking here
> is because if we take this path, we'd have to have a way to provide
> git master already built to the apps and having this on pagure, for
> sure, doesn't seem like a good idea.
>
>>
>>> - Provide a DBus API for osinfo-db-import and osinfo-db-validate (at least):
>>>   With this we can easily spawn, for instance, `osinfo-db-import
>>> https://releases.pagure.org/libosinfo/osinfo-db-latest.tar.xz` from
>>> any app using osinfo-db;
>>
>> I'm pretty sceptical about need for adding dbus into this. I don't see
>> why an app couldn't just spawn the commands as is. Turning them into
>> a DBus service feels like massive overkill for the problem.
>
> Okay from my side.
> I'd like to hear here from Felipe as well why a dbus API would be
> their preferred way and, mainly, whether we could go for spawning the
> command as they are now.
>
>>
>> Also we must not have any applications hardcoding releases.pagure.org
>> URLs. I do not have confidence in pagure.org sticking around over the
>> long time frames RHEL lives for.
>>
>>> Ideas? Possible limitations?
>>
>> I've been thinking about live updates for quite a while - ever since I
>> created the osinfo-db-tools separate package in fact :-)
>>
>> The two key problems with any live update mechanism are scalability and
>> compatibility. We need to ensure that applications don't repeatedly
>> hit the web server to download unchanged content over & over. We also
>> neeed to ensure that we don't bake in usage of a URL that is liable
>> to change over time. Whatever we do will end up in RHEL releases that
>> can live for 10+ years, and I don't have confidence our hosting will
>> live for 10+ years without moving at least once.
>>
>> My thought was to leverage DNS as a key part of the solution.  Have DNS
>> TXT record exposing the current live update URL ie a TXT query against
>> "dbupdate.libosinfo.org"  could return something akin to
>>
>>   url=https://releases.pagure.org/libosinfo/osinfo-db-20180612.tar.xz,version=20180612
>>
>> The locally install osinfo-db content always creates a file VERSION in
>> the root of the install location. So that content can be compared against
>> the DNS result to see if it is outdated of not, and thus avoid hitting
>> the web server at all unless there's genuinely a change that needs
>> acquiring.
>>
>> I would anticipate some osinfo-db-update-check tool to perform this work
>> and print out a URL, that could then be given to osinfo-db-import.
>
> I like the idea!
>
> Who's the owner of libosinfo.org? Daniel? Red Hat?
>
> Daniel,
> Did you start poking at this in the past? If not, it's something I can
> slowly starting to take a look at.
>
>>
>> It feels like a cron job to kick off periodic updates would be nice, but
>> the downside with that is that it synchronizes a DOS attack from all
>> deployed hosts (modulo timezone differences).
>
> The idea about what will be used on Boxes is something to be discussed
> during its hackfest that will  happen next Monday.
> I'm expecting that for generating the flatpack a cron job will be
> used. I'm not sure whether we want to also have an option to update
> the osinfo-db on non-flatpak'ed apps (aka, the ones normally shipped
> on OSes), although I do believe GNOME Boxes should!
>
>>
>> Security is, however, a key issue here. For this to be practical at the
>> very least it would likely require either DNSSEC, or embedding a crypto
>> signature in the result. The latter would however require a long term
>> pub key to be embedded in the client, and I'm not a fan of that.
>>
>> There is a concern of reinventing the wheel here. There is some kind of
>> industry standard I vaguely recall for doing software/content updates
>> over the internet. I've not investigated that in enough detail to
>> understand if it avoids the scalability & compatibility concerns I
>> have though
>
> Hmm. Right, it makes sense. I'll check what the standard is and how we
> could take advantage of that.
> .
>>
>> Regards,
>> Daniel
>> --
>> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
>> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
>> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
>
>
> Best Regards,
> --
> Fabiano Fidêncio



-- 
Fabiano Fidêncio