Ubuntu system image

Add support in system-image server for importing files from a remote system-image instance

Bug #1447633 reported by Daniel Manrique
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ubuntu system image
Triaged
High
Barry Warsaw

Bug Description

Hello,

Right now, there doesn't seem to be an immutable URL pointing to the latest tarballs for a given channel/device. This means, for instance, that if I want to get the latest ubuntu tarball for ubuntu-touch/devel-proposed for a mako device, I have to figure it out with some logic from contents in https://system-image.ubuntu.com/ubuntu-touch/devel-proposed/mako/ (parsing the index.json for example).

This complicates the life of some development system-image-server instances because we can't simply point file_ubuntu to use http generator and a fixed URL. One of these instances I know of, does what I described (parsing json) to decide whether there's a new image to fetch, figure out the filenames and fetch it.

Would it be possible to have some sort of direct link to the latest tarballs? so that for instance:

https://system-image.ubuntu.com/ubuntu-touch/devel-proposed/mako/current/

would contain:

ubuntu.tar.xz -> /pool/ubuntu-adc817ef2d9ce1273c5c77bf0cdc935bdb225800050b25d05078dfa79699044a.tar.xz

and so on, for any files comprising the latest image?

Then I could point my system-image instance to:
file_ubuntu = http;https://system-image.ubuntu.com/ubuntu-touch/devel-proposed/mako/current/ubuntu.tar.xz;name=ubuntu

and know that it will get the latest image.

This is needed for development work by some internal Canonical teams, but may also be useful for any developers who want to have a local system-image installation and have it synchronize tarballs with the official system-image server.

Please let me know if you'd like me to expand on why this is needed, or if any other alternatives to achieve this come to mind.

Revision history for this message
Steve Langasek (vorlon) wrote :

Discussed this in realtime today. Rather than modifying the server to expose a 'latest' link, what I would like to see implemented in system-image server is support for a new "remote_si" (bikeshed the name to taste) generator which talks to an existing system-image server remotely, and walks the tree to find the latest image part (device tarball, etc) on a specified channel - using the full channels.json/channels.json.gpg/index.json validation chain that already exists. This lets us cryptographically ensure integrity of the download using client and server code that already exists, rather than requiring deployment of a new server feature on system-image.ubuntu.com that would only be used by the capomastro server and not shared with other clients of system-image.ubuntu.com.

And as a stopgap measure until this is implemented, we've recommended that PES simply mirror the content from cdimage.ubuntu.com via rsync and run the cdimage generator locally for imports.

Changed in ubuntu-system-image:
importance: Undecided → High
assignee: nobody → Barry Warsaw (barry)
status: New → Triaged
summary: - Add a consistent link or way to download the "latest" of a given tarball
- for a channel/device
+ Add support in system-image server for importing files from a remote
+ system-image instance
Revision history for this message
Caio Begotti (caio1982) wrote :

Ok, so, it's a bit embarrassing but we (Daniel) just discovered there is in fact a remote-cdimage generator inside the server code, which apparently works ok on our staging env. Anyway, I believe it would be important if Barry could take a look at that just in case it's doing something too weird or in a non complete fashion as you guys expected.

We're not done yet with this problem but it's just a heads up. We're still testing to see if this lib/systemimage/generators.py:generate_file_remote_system_image is good.

Revision history for this message
Daniel Manrique (roadmr) wrote :

This file configuration goes into system-image and reads the json files to figure out the latest ubuntu tarball to get for the mako device and the ubuntu-touch/devel-proposed channel.

file_ubuntu = remote-system-image;http://system-image.ubuntu.com;ubuntu-touch/devel-proposed;ubuntu;device=mako;name=ubuntu

This is in the generate_file_remote_system_image method. One thing I noticed it doesn't do, comparing with what we discussed it should do, is verify the signatures. All it does is get the json files, starting with the index.json top-level one and working its way down from there.

The tarball will be stored in the pool and named using the checksum from the original file. So it blindly trusts that the checksum is correct (though I think it's only used for file naming, not for actual verification). This is nice because it guarantees that the tarball will be named exactly the same as it was in the upstream system-image.

So functionally this fits the requirement for this bug, but perhaps it's worth fleshing out this generator so it does validation of the chain of trust.

Revision history for this message
Steve Langasek (vorlon) wrote :

IMHO we shouldn't consider this bug resolved until there is gpg verification of the downloads.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.