Newlines in Sources.bz2 indices

Downloading http://ppa.launchpad.net/moblin/ppa/ubuntu/dists/karmic/main/source/Sources.bz2 file ...
Decompressing http://ppa.launchpad.net/moblin/ppa/ubuntu/dists/karmic/main/source/Sources.bz2 file ...
Traceback (most recent call last):
  File "/usr/bin/germinate-update-metapackage", line 273, in <module>
    germinator, [dist], components, architecture, cleanup=True)
  File "/usr/lib/germinate/Germinate/Archive/tagfile.py", line 121, in feed
    "source/Sources"))
  File "/usr/lib/germinate/Germinate/germinator.py", line 323, in parseSources
    ver = p.Section["Version"]
KeyError: 'Version'

It seems we have accepted some DSC with this format mistake:

{{{
launchpad_prod=> SELECT spn.name as name, spr.version as version, o.name as owner, a.name as archive from sourcepackagerelease spr, sourcepackagename spn, archive a, person o where spr.upload_archive=a.id AND a.owner=o.id AND spr.sourcepackagename=spn.id AND dsc_binaries LIKE '%
%';
      name | version | owner | archive
----------------+-------------------------+----------------+---------
 openoffice.org | 1:3.1.1-2ubuntu1 | ubuntu-drivers | primary
 linux | 2.6.31-10.36~undervolt1 | raaa | testing
 mono | 2.4.2.3+dfsg-2 | ubuntu-drivers | primary
 linux | 2.6.31-10.36~undervolt2 | raaa | testing
 openoffice.org | 1:3.1.1-2ubuntu1moblin1 | moblin | ppa
 linux | 2.6.31-10.35~awe1 | awe | ppa
 linux | 2.6.31-10.36~eee1 | yofel | ppa
 linux | 2.6.31-10.36~undervolt3 | raaa | testing
 linux | 2.6.31-10.36~eee2 | yofel | ppa
 linux | 2.6.31-10.36~eee3 | yofel | ppa
 linux | 2.6.31-10.36~undervolt4 | raaa | testing
 openoffice.org | 1:3.1.1-2ubuntu1moblin2 | moblin | ppa
(12 rows)

This is somehow related with bug #435316, we have to find the right way to identify this format problem (I'm assuming it is a problem) and reject those uploads.

Since there are not *many* yet, the migration of the existing that will be easy.

From the Debian (and Ubuntu) Policy Manual:

5.6.19. `Binary'
----------------

     This field is a list of binary packages. Its syntax and meaning
     varies depending on the control file in which it appears.

     When it appears in the `.dsc' file, it lists binary packages which a
     source package can produce, separated by commas[1]. It may span
     multiple lines.

This is preventing OEM Services from running germinate for some Intel work we're doing. Very important this gets fixes ASAP.

Just trying to reproduce this locally - with a pristine sources.txt in the same directory in a python console:

{{{
f = open('sources.txt')
import apt_pkg
p = apt_pkg.ParseTagFile(f)
while p.Step() == 1:
   ver = p.Section['Package']
   src = p.Section['Version']

f.close()
}}}

Add *two* consecutive new-lines within the Binary: section as you like and save, then:

{{{
f = open('sources.txt')
p = apt_pkg.ParseTagFile(f)
while p.Step() == 1:
    src = p.Section['Package']
    ver = p.Section['Version']

}}}

The two consecutive new-lines results in:
Traceback (most recent call last):
  File "<console>", line 3, in ?
KeyError: 'Version'

But using just one new-line (as described in the bug) seems to work fine. This is with apt_pkg.Version == '0.7.20.2ubuntu6'.

I'll try to find out which version of apt-pkg is being used and verify the problem.

OK, exactly the same (two new-lines required) to replicate this on dogfood which has:

$ dpkg-query -W apt python-apt
apt 0.7.9ubuntu17.5~0.CAT.8.04
python-apt 0.7.4ubuntu7.5

which, after checking with cjwatson is the same as the production publisher.

So it seems when the original dsc contains a single \n, we're publishing a Sources.bz2 with a *blank* line, which is incorrect and apt_pkg correctly fails to parse it.

OK, now for the fix...

The actual issue is in lp.archiveuploader.parse_tagfile_lines - it has introduce a limited number of corrupted values for SourcePackageRelease.dsc_binaries with a trailing '\n'.

I've created and tested an intermediate work-around that rstrips the dsc_binaries when creating the index file, which will give us a bit more time to solve the original issue properly.

Fixed in devel r9606 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/9606>

I'm setting the status back to Triaged, as what we've committed is a work-around - not a fix. It's a work-around that we can cherry-pick if necessary, and it will ensure that there are no trailing '\n' at end of fields in the Sources index file, so that the file can be correctly parsed by apt_pkg.ParseTagFile().

The work-around does not ensure that the stray trailing '\n' will not get into the SPR.dsc_binaries field in the first place, nor does it fix the corrupt data. Hence leaving this bug open.

I've not verified it yet, but we're 90% certain that when there are valid single '\n' chars within a field's values (as in the snippet in the bug description above), the lp.archiveuploader.parse_tagfile_lines is appending a trailing '\n' to the end of the value (also shown in the snippet above) - and it is this trailing '\n' that is invalid and causing the problem (If/when I have read access, we can simply run a slightly modified version of Celso's above query with the condition '%\n' to verify that these SPRs all have an invalid trailing '\n'.

I can't see any reason why we are not simply using apt_pkg.ParseTagFile() instead of our own parse_tagfile_lines() method, so I'd be keen to replace it if there is no reason not to. Celso's thought was that, as we are seeing more and more debian packages with the valid '\n' coming through, this issue could snow-ball, meaning more corrupt data for which we are relying on the work-around.

As another example, currently the Sources in the following ppa (identified by Celso's query above) displays the problem also:
http://ppa.launchpad.net/awe/ppa/ubuntu/dists/karmic/main/source/Sources
{{{
Package: linux
Binary: linux-source-2.6.31, linux-doc, linux-headers-2.6.31-10, linux-libc-dev, linux-image-2.6.31-10-386, linux-headers-2.6.31-10-386, linux-image-debug-2.6.31-10-386, linux-image-2.6.31-10-generic, linux-headers-2.6.31-10-generic, linux-image-debug-2.6.31-10-generic, linux-image-2.6.31-10-generic-pae, linux-headers-2.6.31-10-generic-pae, linux-image-debug-2.6.31-10-generic-pae, linux-image-2.6.31-10-ia64, linux-headers-2.6.31-10-ia64, linux-image-debug-2.6.31-10-ia64, linux-image-2.6.31-10-lpia, linux-headers-2.6.31-10-lpia, linux-image-debug-2.6.31-10-lpia, linux-image-2.6.31-10-powerpc, linux-headers-2.6.31-10-powerpc, linux-image-debug-2.6.31-10-powerpc, linux-image-2.6.31-10-powerpc64-smp, linux-headers-2.6.31-10-powerpc64-smp, linux-image-debug-2.6.31-10-powerpc64-smp, linux-image-2.6.31-10-powerpc-smp, linux-headers-2.6.31-10-powerpc-smp, linux-image-debug-2.6.31-10-powerpc-smp, linux-image-2.6.31-10-server, linux-headers-2.6.31-10-server, linux-image-debug-2.6.31-10-server,
linux-image-2.6.31-10-sparc64, linux-headers-2.6.31-10-sparc64, linux-image-debug-2.6.31-10-sparc64, linux-image-2.6.31-10-sparc64-smp, linux-headers-2.6.31-10-sparc64-smp, linux-image-debug-2.6.31-10-sparc64-smp, linux-image-2.6.31-10-virtual, kernel-image-2.6.31-10-generic-di, nic-modules-2.6.31-10-generic-di, nic-shared-modules-2.6.31-10-generic-di, serial-modules-2.6.31-10-generic-di, ppp-modules-2.6.31-10-generic-di, firewire-core-modules-2.6.31-10-generic-di, scsi-modules-2.6.31-10-generic-di, plip-modules-2.6.31-10-generic...

Fixed in devel r9708 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/revision/9708>