disk corruption in release/distfiles/experimental mirror system

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

disk corruption in release/distfiles/experimental mirror system

Robin H. Johnson-2
Hi everybody,

We've potentially traced down some disk corruption to one of the
masterdistfiles.gentoo.org servers that we use to distribute distfiles/,
releases/ & experimental/ to mirrors. The Gentoo repo of ebuilds does
NOT go via these systems, and is unaffected.

It's NOT on the source box where we have the content, and we don't have
any direct access to these systems ourselves (despite the hostname).

We're waiting for all of the DIGESTS to be checked on that mirror now,
and then we'll potentially double-bump the timestamps to get mirrors to
update the files (first to a new current timestamp, followed by going
back to the original timestamp).

If you'd like to check your own mirrors, WITHOUT having to use rsync
--checksum, we're pushing out a helper script in the releases directory.
It'll be at releases/verify-digests.sh, or your can grab it from Git:
https://gitweb.gentoo.org/infra/mastermirror-scripts.git/tree/verify-digests.sh
It does run with ionice at idle priority, and can check everything or
just specific files as needed.

This was brought to our attention per bug 581924, bug 581638, amongst
others.

Known affected files to date are:
distfiles/chromium-50.0.2661.94.tar.xz
releases/amd64/autobuilds/20160428/install-amd64-minimal-20160428.iso
releases/amd64/autobuilds/20160602/install-amd64-minimal-20160602.iso
releases/x86/autobuilds/20160531/stage3-i686-20160531.tar.bz2

The first two have expired out of the system already due to being old,
but the other files are still present.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

signature.asc (455 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: disk corruption in release/distfiles/experimental mirror system

nholland
On Thu, June 9, 2016 11:22 pm, Robin H. Johnson wrote:

> Known affected files to date are:
> distfiles/chromium-50.0.2661.94.tar.xz
> releases/amd64/autobuilds/20160428/install-amd64-minimal-20160428.iso
> releases/amd64/autobuilds/20160602/install-amd64-minimal-20160602.iso
> releases/x86/autobuilds/20160531/stage3-i686-20160531.tar.bz2

What about
experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2?

This file failed digest verification when I let your script run on my
mirror. I then grabbed a copy from masterdistfiles.gentoo.org, which also
fails verification (and yields the same digest as my copy of the file
did). I then tried unpacking the file, and sure enough, the unpacking
process aborts with an error, too.

Now, this file is from the "experimental" and not from the "releases"
tree, but I guess it'd still be nice if the file wasn't corrupt.

Greetings
Nils


Reply | Threaded
Open this post in threaded view
|

Re: disk corruption in release/distfiles/experimental mirror system

nholland
In reply to this post by Robin H. Johnson-2
On Thu, June 9, 2016 11:22 pm, Robin H. Johnson wrote:

> Known affected files to date are:
> distfiles/chromium-50.0.2661.94.tar.xz
> releases/amd64/autobuilds/20160428/install-amd64-minimal-20160428.iso
> releases/amd64/autobuilds/20160602/install-amd64-minimal-20160602.iso
> releases/x86/autobuilds/20160531/stage3-i686-20160531.tar.bz2

And then, besides the experimental MIPS file I wrote about a few hours
ago, there's also

releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2

which was corrupt on my mirror and also proved to be corrupt when being
fetched from masterdistfiles. In addition to the files you already named,
these were the only two corrupt files I spotted on my mirror, so with a
little luck, these are the only two additional corruptions that you should
find during your digest verification on masterdistfiles as well.

Stuff in distfiles/ hasn't been checked here (yet)...

Greetings
Nils


Reply | Threaded
Open this post in threaded view
|

Re: disk corruption in release/distfiles/experimental mirror system

Robin H. Johnson-2
On Sun, Jun 12, 2016 at 04:22:38PM -0000, Nils Holland wrote:
> experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2
> releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2
>
> which was corrupt on my mirror and also proved to be corrupt when being
> fetched from masterdistfiles. In addition to the files you already named,
> these were the only two corrupt files I spotted on my mirror, so with a
> little luck, these are the only two additional corruptions that you should
> find during your digest verification on masterdistfiles as well.
Can you please post:
- The IP(s) for masterdistfiles that you got those broken files from [1].
- The checksums of the broken files.

> Stuff in distfiles/ hasn't been checked here (yet)...
Tool for transforming the Portage Manifests into a usable digests file
in the works still; but it won't cover everything right away (due to
distfiles for prefix, overlays, other retention requirements).

[1] Depending on GeoDNS to your location, there are 4+ different IPs you
could hit for masterdistfiles.g.o.

--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : [hidden email]
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

Reply | Threaded
Open this post in threaded view
|

Re: disk corruption in release/distfiles/experimental mirror system

nholland
On Sun, Jun 12, 2016 at 06:37:48PM +0000, Robin H. Johnson wrote:

> Can you please post:
> - The IP(s) for masterdistfiles that you got those broken files from [1].
> - The checksums of the broken files.

Right, so masterdistfiles.gentoo.org resolves to the two IP addresses
of 64.50.236.52, 64.50.233.100 here. Consequently, I just now fetched
the file

/experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2

(to test one of the files I spotted yesterday) once from both IPs, and
in both cases, they yield the sha512sum of

ab01d59eebc811789d1ca592da4330e34233832f07ebaa9808bb12a24b56a1a694a657ad817b7e8de30cd031da6f630d2cf41ad5066e09692b791efefe8960dd

Correct, however, according to the digest files, is

758d304df8b24a842466713ee5297b62bc66674041ed462a1f8374a6281455b283411395cccd318b31edc2a3ce289f475fb090eb418bec6a2e1bdd3b4aee4610

Also...:

nils@boerne ~ $ bzip2 -tvv stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2
  stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2:
  [1: huff+mtf rt+rld]
  [2: huff+mtf rt+rld]
  [3: huff+mtf rt+rld]
  [4: huff+mtf rt+rld]
  [...]
  [330: huff+mtf rt+rld]
  [331: huff+mtf rt+rld]
  [332: huff+mtf rt+rld]data integrity (CRC) error in data

I didn't re-check the other broken files, but chances are I'd get the
same results I reported yesterday for these files as well.

Greetings
Nils

Reply | Threaded
Open this post in threaded view
|

Re: disk corruption in release/distfiles/experimental mirror system

nholland
On Mon, Jun 13, 2016 at 09:33:52AM +0200, Nils Holland wrote:

> I didn't re-check the other broken files, but chances are I'd get the
> same results I reported yesterday for these files as well.

Addendum:

releases/amd64/autobuilds/20160602/hardened/stage4-amd64-hardened+cloud-20160602.tar.bz2

seems to be correct when fetched from masterdistfiles at eiter
64.50.236.52 or 64.50.233.100, and is only broken on some other
mirrors (mine included). I could swear, however, that yesterday it was corrupt
on masterdistfiles as well, though.

/experimental/mips/uclibc/mips32r2/stage3-mips32r2-uclibc-vanilla-20160410.tar.bz2

is however, like I said in my previous mail, indeed still problematic
when fetched from 64.50.236.52 or 64.50.233.100; I just tested that a
few minutes ago.

Greetings
Nils