gemato in prefix

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

gemato in prefix

Michael Weiser
Hi,

does it make sense to look into using gemato for repo verification or is
there a reason this cannot work currently?
--
Thanks,
Michael

Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Fabian Groffen-2
It should, but I didn't get around to it.  Instead I want to use my own
C-based tool, but I also didn't get around to getting it ready.

Fabian


On 02-02-2018 14:27:52 +0100, Michael Weiser wrote:
> Hi,
>
> does it make sense to look into using gemato for repo verification or is
> there a reason this cannot work currently?
> --
> Thanks,
> Michael
>

--
Fabian Groffen
Gentoo on a different level

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Michael Weiser
Hi Fabian,

On Fri, Feb 02, 2018 at 09:06:34PM +0100, Fabian Groffen wrote:

> > does it make sense to look into using gemato for repo verification or is
> > there a reason this cannot work currently?

> It should, but I didn't get around to it.

I finally got around to trying gemato on my Mac. It sets off fine but
immediately fails on sys-apps/Manifest.gz:

$ gemato verify -K /Users/michael/b/pubring.gpg /usr/local/gentoo/usr/portage/
INFO:root:Refreshing keys from keyserver...
INFO:root:Keys refreshed.
INFO:root:Manifest timestamp: 2018-02-19 17:28:21 UTC
INFO:root:Valid OpenPGP signature found:
INFO:root:- primary key: 0204A8ABD003E57A9558850DBA08091EC6317B3C
INFO:root:- subkey: 0204A8ABD003E57A9558850DBA08091EC6317B3C
INFO:root:- timestamp: 2018-02-19 17:28:21 UTC
INFO:root:Verifying /usr/local/gentoo/usr/portage/...
ERROR:root:Manifest mismatch for sys-apps/Manifest.gz
  BLAKE2B: expected: 304895d779741fedeaac05df18857d5b0c1afa23220e6e578bd7ddca53f6d4781751881f13c59c361f3a225c7c8290cfa2ae278c779ad4c68a938b9336ebc999, have: e2260c115b7886ac16e74a8c981f3830650c018aa2d1566947b7eee2463eb8c56a5d5be3d30f324b239b3f9899b1781fe6f11c3bfb482bebb8df48e09e15ef43
  SHA512: expected: 0985d753fcb39735651606c30dbe9335d6d82569ca0e6ac766f268f5fd8d3df40e9f2664c145c752bb9c7c09a06f7766bc9fdb42a37809e62ea6462743bde2c6, have: 9d60081f638b5678780c21f698f0ee56cd4fa4dfe3d89a6c38403a37bd6cd782181fe0368af597d316f110e82c61cc8770346007a2a63dad90b7bac555c277eb

I can reproduce the discrepancy with sha512sum and b2sum.

Is it possible that prefix's tree isn't fully rehashed and resigned
after changes?

> Instead I want to use my own
> C-based tool, but I also didn't get around to getting it ready.

Is it available somewhere to try out?
--
Thanks, Michael

Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Fabian Groffen-2
Well, yeah, I have the feeling that until I'm done with the verification
(it's a work in progress, the problem is mostly in walking the entire
tree a bit efficient) I can see if what we have actually makes sense.

Thing is I once believed Portage checked manifest and all, but it seems
not to do anything any more, so my idea of things being OK may have been
false appearances because Portage no longer gives a ****.

Fabian


On 20-02-2018 20:25:33 +0100, Michael Weiser wrote:

> Hi Fabian,
>
> On Fri, Feb 02, 2018 at 09:06:34PM +0100, Fabian Groffen wrote:
>
> > > does it make sense to look into using gemato for repo verification or is
> > > there a reason this cannot work currently?
>
> > It should, but I didn't get around to it.
>
> I finally got around to trying gemato on my Mac. It sets off fine but
> immediately fails on sys-apps/Manifest.gz:
>
> $ gemato verify -K /Users/michael/b/pubring.gpg /usr/local/gentoo/usr/portage/
> INFO:root:Refreshing keys from keyserver...
> INFO:root:Keys refreshed.
> INFO:root:Manifest timestamp: 2018-02-19 17:28:21 UTC
> INFO:root:Valid OpenPGP signature found:
> INFO:root:- primary key: 0204A8ABD003E57A9558850DBA08091EC6317B3C
> INFO:root:- subkey: 0204A8ABD003E57A9558850DBA08091EC6317B3C
> INFO:root:- timestamp: 2018-02-19 17:28:21 UTC
> INFO:root:Verifying /usr/local/gentoo/usr/portage/...
> ERROR:root:Manifest mismatch for sys-apps/Manifest.gz
>   BLAKE2B: expected: 304895d779741fedeaac05df18857d5b0c1afa23220e6e578bd7ddca53f6d4781751881f13c59c361f3a225c7c8290cfa2ae278c779ad4c68a938b9336ebc999, have: e2260c115b7886ac16e74a8c981f3830650c018aa2d1566947b7eee2463eb8c56a5d5be3d30f324b239b3f9899b1781fe6f11c3bfb482bebb8df48e09e15ef43
>   SHA512: expected: 0985d753fcb39735651606c30dbe9335d6d82569ca0e6ac766f268f5fd8d3df40e9f2664c145c752bb9c7c09a06f7766bc9fdb42a37809e62ea6462743bde2c6, have: 9d60081f638b5678780c21f698f0ee56cd4fa4dfe3d89a6c38403a37bd6cd782181fe0368af597d316f110e82c61cc8770346007a2a63dad90b7bac555c277eb
>
> I can reproduce the discrepancy with sha512sum and b2sum.
>
> Is it possible that prefix's tree isn't fully rehashed and resigned
> after changes?
>
> > Instead I want to use my own
> > C-based tool, but I also didn't get around to getting it ready.
>
> Is it available somewhere to try out?
> --
> Thanks, Michael
>
--
Fabian Groffen
Gentoo on a different level

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Michael Weiser
Hi Fabian,

On Tue, Feb 20, 2018 at 08:41:57PM +0100, Fabian Groffen wrote:

> Thing is I once believed Portage checked manifest and all, but it seems
> not to do anything any more, so my idea of things being OK may have been

I also was a bit surprised to find that portage didn't authenticate and
verify the tree at all. Stumbling over webrsync more or less by
accident, I've been using it as the next best thing in the interim.

From what I was able to find on the net, there's never been any
actual implementation before Michal Gorny started gemato (see
https://www.gentoo.org/glep/glep-0074.html#motivation wrt GLEP-58 from
2008 never being implemented). After using gemato on Gentoo Linux as a
very early adopter I'm eager to get something comparable going in Prefix
Mac.

How can I help?
--
Thanks,
Michael

Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Fabian Groffen-2
Hi Michael,

To give you an idea; the current tree is getting its Manifests from
hashgen.c, which you can find in scripts/rsync-generation/hashgen.c.
The hashverify tool, which I'm currently working on, is basically an
addition to that file (doing argv[0] detection) to perform the
verification.  At this time of writing, I have the gpg-verification and
single file entry verification in place.  I'm still trying to close the
gap in checking the dirs in particular looking for files that are not
listed in the Manifest.
hashgen currently runs in 30s or so on the tree to generate manifests.
I hope it can verify in the same amount of time (we're talking about a
Quad G5 PowerPC machine here with rusty old spinning disks), leaving it
in a much better position to be used for Prefix, since we tend to have
slower/older machines around.  We're not really looking forward to 15
minutes of verification as some bugs have been reported to with gemato.

Portage used to do 1) checking the digests of ebuilds and 2) checking
for missing and extra files.  I noticed that at least 1) is no longer
present, which I find weird.  I need checking this on normal Gentoo,
(simply edit an ebuild and try to emerge it without updating its
digest), but I have the suspicion this got disabled because the full
tree verification should catch this.  Needless to say, that's
suboptimal, and not very secure IMO.

Now I may be all wrong with trying to implement the verification myself,
but that's a separate topic.  gemato should work fine.

I've checked some of the digests you mentioned and they look ok.  So I'm
wondering whether perhaps you got caught in the middle of a sync.  This
used to be much less of a problem because of per-ebuild-dir integrity,
but now the entire tree requires to be consistent.  I'll look into
re-activating my symlink-flip, which should make the switch atomic, but
I don't know what rsync is doing if the symlink is flipped during a
sync.  It reduces the invalid window somewhat I guess.

Thanks,
Fabian


On 20-02-2018 21:24:03 +0100, Michael Weiser wrote:

> Hi Fabian,
>
> On Tue, Feb 20, 2018 at 08:41:57PM +0100, Fabian Groffen wrote:
>
> > Thing is I once believed Portage checked manifest and all, but it seems
> > not to do anything any more, so my idea of things being OK may have been
>
> I also was a bit surprised to find that portage didn't authenticate and
> verify the tree at all. Stumbling over webrsync more or less by
> accident, I've been using it as the next best thing in the interim.
>
> From what I was able to find on the net, there's never been any
> actual implementation before Michal Gorny started gemato (see
> https://www.gentoo.org/glep/glep-0074.html#motivation wrt GLEP-58 from
> 2008 never being implemented). After using gemato on Gentoo Linux as a
> very early adopter I'm eager to get something comparable going in Prefix
> Mac.
>
> How can I help?
> --
> Thanks,
> Michael
>
--
Fabian Groffen
Gentoo on a different level

signature.asc (201 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gemato in prefix

Michael Weiser
Hi Fabian,

On Wed, Feb 21, 2018 at 10:04:42AM +0100, Fabian Groffen wrote:

> To give you an idea; the current tree is getting its Manifests from
> hashgen.c, which you can find in scripts/rsync-generation/hashgen.c.
> The hashverify tool, which I'm currently working on, is basically an
> addition to that file (doing argv[0] detection) to perform the
> verification.  At this time of writing, I have the gpg-verification and
> single file entry verification in place.  I'm still trying to close the
> gap in checking the dirs in particular looking for files that are not
> listed in the Manifest.

I've been following or rather post-reading the discussion about GLEP-74
and it seems a lot of thought and security considerations have gone into
it. Are you following it for your implementation?

I see you're using OpenMP. That looks very neat and could certainly make
it scale better than gemato.

BTW: I remember OpenMP being plain missing from clang until recently.
Ah, https://clang-omp.github.io/ says 3.7 and onwards have native
support.

> hashgen currently runs in 30s or so on the tree to generate manifests.
> I hope it can verify in the same amount of time (we're talking about a
> Quad G5 PowerPC machine here with rusty old spinning disks), leaving it
> in a much better position to be used for Prefix, since we tend to have
> slower/older machines around.

I can only believe that number if the portage tree fits into the fs
cache (RAM). Then it would be purely CPU-bound, I guess, and C could
play out its advantage.

On a machine that has just synced the tree using rsync it would also
still be in RAM as a working set (if there was sufficient RAM) and the
same consideration would apply. On my MacBook Air I get 625012k from du
for the tree. So this should be alright for any machine with 1GB+ of RAM
that's not doing anything else at the time.

As soon as the tree doesn't completely fit into RAM, the later files of
the rsync will push the first files out of the cache. So for
verification, the reading will start from scratch, making the
verification I/O-bound and the advantage of a C implementation mostly
irrelevant, won't it?

As a quick number: On an ARM SBC of mine with a Dual-Core 1.2GHz
Cortex-A7, 1GB of RAM and fs on a microSD card that does 22MB/s
sequential read I get 400 seconds flat for the first run of gemato
verify and 386 for the second when the fs cache is primed already. I do
see a bit of I/O-boundness on the first run and on the second, gemato
fully hogs the CPU. It would be interesting to compare how much of that
is python overhead and how much the actual crypto. Unfortunately the gcc
on that board doesn't have openmp support (yet).

gemato create runs 667s on the same board and falls back to a single
thread for some reason about a minute in. So I'd certainly take a
22fold speedup if I could get it. :)

> We're not really looking forward to 15
> minutes of verification as some bugs have been reported to with gemato.

Well, with webrsync I was looking at around three minutes of download
and five minutes of unpacking and local rsync on my ARM SBCs. With
gemato it's now about 30 seconds to three minutes of rsyncing depending
on the amount of changes and six minutes of gemato checking. So
basically I've neither gained nor lost anything but feel much more
efficient by once again not downloading what hasn't changed.

It also shows that I have a high pain threshold in dealing with Gentoo
on small and old machines. Considering how long Gentoo users typically
wait for compiles to finish I wouldn't have expected a bit of a wait for
gemato or hashverify to be much of an issue.

> Portage used to do 1) checking the digests of ebuilds and 2) checking
> for missing and extra files.  I noticed that at least 1) is no longer
> present, which I find weird.  I need checking this on normal Gentoo,
> (simply edit an ebuild and try to emerge it without updating its
> digest), but I have the suspicion this got disabled because the full
> tree verification should catch this.  Needless to say, that's
> suboptimal, and not very secure IMO.

Ah, now I understand what you were getting at. I just verified: Both
checks are working on Gentoo Linux but not in Prefix Mac. It seems to
have to do with setting "thin-manifests" in layout.conf of the prefix
tree. Once I set that to false, prefix portage behaves the same as
Gentoo Linux portage.

Gentoo Linux:

[root@linux:/usr/portage/xfce-base/garcon] sha512sum garcon-0.6.1.ebuild
dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7 garcon-0.6.1.ebuild
[root@linux:/usr/portage/xfce-base/garcon] grep EBUILD.*0.6.1.*SHA512 Manifest
EBUILD garcon-0.6.1.ebuild 1018 SHA512 dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7
[root@linux:/usr/portage/xfce-base/garcon] sed -i -e "s,econf,fooconf," garcon-0.6.1.ebuild
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild compile
 * Digest verification failed:
 * /usr/portage/xfce-base/garcon/garcon-0.6.1.ebuild
 * Reason: Filesize does not match recorded size
 * Got: 1020
 * Expected: 1018

[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild digest
>>> Creating Manifest for /usr/portage/xfce-base/garcon
[root@linux:/usr/portage/xfce-base/garcon] mkdir files
[root@linux:/usr/portage/xfce-base/garcon] echo foo > files/foo
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild prepare
 * garcon-0.6.1.tar.bz2 BLAKE2B SHA512 size ;-) ...
[ ok ]
 * checking ebuild checksums ;-) ...
[ ok ]
 * checking miscfile checksums ;-) ...
[ ok ]
!!! A file is not listed in the Manifest:
'/usr/portage/xfce-base/garcon/files/foo'
[root@linux:/usr/portage/xfce-base/garcon] touch garcon-0.7.0.ebuild
[root@linux:/usr/portage/xfce-base/garcon] ebuild garcon-0.6.1.ebuild prepare
 * A file is not listed in the Manifest: '/usr/portage/xfce-base/garcon/garcon-0.7.0.ebuild'
[root@linux:~] grep thin /usr/portage/metadata/layout.conf
# Use thin Manifests for Git
thin-manifests = false

Prefix Mac:

root@mac:/gentoo/usr/portage/xfce-base/garcon $ sha512sum garcon-0.6.1.ebuild
dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7 garcon-0.6.1.ebuild
root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep EBUILD.*0.6.1.*SHA512 Manifest
EBUILD garcon-0.6.1.ebuild 1018 SHA512 dfa80c3e8c766af3d170536f6d7c48793da6bccd4f1647ac8194130a521eb852bf6872514a75e8da7d0eec02eca53d76e431c4371c6f9c891a68fa516fdca8b7
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,econf,fooconf," garcon-0.6.1.ebuild
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sha512sum garcon-0.6.1.ebuild
42f04d921b82955f8ac6237b4f2ebd78fad1cb0f2ab3cbf0f11b4b055b5a20c0cba6f2a37639b759a8912ec728bbde2f491b57d7565066f3ef5f4587e8f787a5 garcon-0.6.1.ebuild
root@mac:/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild compile
>>> Downloading 'http://distfiles.gentoo.org/distfiles/garcon-0.6.1.tar.bz2'
[...]

root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep -r thin /gentoo/usr/portage/metadata/layout.conf
# Use thin Manifests for Git
thin-manifests = true
root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,thin-manifests = true,thin-manifests = false," /gentoo/usr/portage/metadata/layout.conf

root@mac:/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild prepare
>>> Existing ${T}/environment for 'garcon-0.6.1' will be sourced. Run
>>> 'clean' to start with a fresh environment.
 * Missing digest for '/gentoo/usr/portage/xfce-base/garcon/garcon-0.6.1.ebuild'

root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ mkdir files
root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ touch files/foo
root@nindamos:/usr/local/gentoo/usr/portage/xfce-base/garcon $ ebuild garcon-0.6.1.ebuild prepare
 * garcon-0.6.1.tar.bz2 BLAKE2B SHA512 size ;-) ...
[ ok ]
 * checking ebuild checksums ;-) ...
[ ok ]
 * checking miscfile checksums ;-) ...
[ ok ]
!!! A file is not listed in the Manifest: '/usr/local/gentoo/usr/portage/xfce-base/garcon/files/foo'

> Now I may be all wrong with trying to implement the verification myself,
> but that's a separate topic.  gemato should work fine.

I think it's a worthy cause just for the fact that multiple
implementations of a protocol are always good to drive improvement and
show up the corner cases.

Maybe it shouldn't be hidden away in the prefix tree so others can more
easily become aware of and contribute to it.

> I've checked some of the digests you mentioned and they look ok.  So I'm
> wondering whether perhaps you got caught in the middle of a sync.  This
> used to be much less of a problem because of per-ebuild-dir integrity,
> but now the entire tree requires to be consistent.  I'll look into
> re-activating my symlink-flip, which should make the switch atomic, but
> I don't know what rsync is doing if the symlink is flipped during a
> sync.  It reduces the invalid window somewhat I guess.

I've retried a couple of times today. I deleted timestamp.chk several
times and resynced until I saw no delta any more. There actually was a
delta on the second and sometimes third run, mostly Manifest.gzs and
metadata. It still fails reliably on second-level Manifests, e.g.
dev-dotnet/Manifest.gz and sys-libs/Manifest.gz

It looks as if regeneration of Manifests and md5-cache runs almost
continuously, not just every 26,56 of the hour as rsync's motd says. Is
that intentional or are perhaps multiple instances of the script running
amok?
--
Micha
Elephants don't play chess!

Reply | Threaded
Open this post in threaded view
|

Re: tree verification in prefix

Fabian Groffen-2
Please excuse my snipping.

On 21-02-2018 21:55:29 +0100, Michael Weiser wrote:
> I've been following or rather post-reading the discussion about GLEP-74
> and it seems a lot of thought and security considerations have gone into
> it. Are you following it for your implementation?

Sure.  I have my reservations about the decisions now, but you only get
that much of detail when you start implementing such spec.

> I see you're using OpenMP. That looks very neat and could certainly make
> it scale better than gemato.

That was because I was too lazy to do pthreads stuff this time, and at
the time of writing hashgen was experimenting with OpenMP as new thing I
learnt.  The target platform rsync0 ran on (Solaris-x64 back then,
ppc64-linux now, it's been amd64-linux inbetween) always supported
OpenMP, so I could unconditionally use it.

> BTW: I remember OpenMP being plain missing from clang until recently.
> Ah, https://clang-omp.github.io/ says 3.7 and onwards have native
> support.

Yes, in the beginning it was an issue with clang on OSX, but the joy of
OpenMP in a way, it just compiles and runs way slower :)

> > hashgen currently runs in 30s or so on the tree to generate manifests.
> > I hope it can verify in the same amount of time (we're talking about a
> > Quad G5 PowerPC machine here with rusty old spinning disks), leaving it
> > in a much better position to be used for Prefix, since we tend to have
> > slower/older machines around.
>
> I can only believe that number if the portage tree fits into the fs
> cache (RAM). Then it would be purely CPU-bound, I guess, and C could
> play out its advantage.

Sure.  The ppc64 box I mention has 16GiB of ram, and 4 cores 2.5GHz
each.  14GiB of ram is used for buffers, so I may very well be a bit
over-optimistic here.

> It also shows that I have a high pain threshold in dealing with Gentoo
> on small and old machines. Considering how long Gentoo users typically
> wait for compiles to finish I wouldn't have expected a bit of a wait for
> gemato or hashverify to be much of an issue.

I have my Sun Blade 100 (sparcv9 502MHz) with 512MiB ram in mind, which
is very slow in any IO tasks, reasonable on CPU bound tasks.

> Ah, now I understand what you were getting at. I just verified: Both
> checks are working on Gentoo Linux but not in Prefix Mac. It seems to
> have to do with setting "thin-manifests" in layout.conf of the prefix
> tree. Once I set that to false, prefix portage behaves the same as
> Gentoo Linux portage.

THANKS.  I'm going to fix this immediately.

> Gentoo Linux:
> [root@linux:~] grep thin /usr/portage/metadata/layout.conf
> # Use thin Manifests for Git
> thin-manifests = false
>
> Prefix Mac:
> root@mac:/gentoo/usr/portage/xfce-base/garcon $ grep -r thin /gentoo/usr/portage/metadata/layout.conf
> # Use thin Manifests for Git
> thin-manifests = true
> root@mac:/gentoo/usr/portage/xfce-base/garcon $ sed -i -e "s,thin-manifests = true,thin-manifests = false," /gentoo/usr/portage/metadata/layout.conf

> > Now I may be all wrong with trying to implement the verification myself,
> > but that's a separate topic.  gemato should work fine.
>
> I think it's a worthy cause just for the fact that multiple
> implementations of a protocol are always good to drive improvement and
> show up the corner cases.

My thinking, plus pet-project material :)

> Maybe it shouldn't be hidden away in the prefix tree so others can more
> easily become aware of and contribute to it.

Once hashverify is done, I'll make an ebuild for it.  hashgen is not
useful in general.  Only if you create a tree yourself.

> I've retried a couple of times today. I deleted timestamp.chk several
> times and resynced until I saw no delta any more. There actually was a
> delta on the second and sometimes third run, mostly Manifest.gzs and
> metadata. It still fails reliably on second-level Manifests, e.g.
> dev-dotnet/Manifest.gz and sys-libs/Manifest.gz

Odd, was this rsync1 or rsync2?

> It looks as if regeneration of Manifests and md5-cache runs almost
> continuously, not just every 26,56 of the hour as rsync's motd says. Is
> that intentional or are perhaps multiple instances of the script running
> amok?

The script is fired off through cron at 26 and 56.  I may have gone
wrong here, but for the sake of redundancy, I scratched the
  rsync0 -> rsync{1,2}
path.  Instead rsync1 and rsync2 are doing full generation both of them.
To make this be equal, I had to play a trick with timestamps, I
basically set the timestamp to the git commit time.  Maybe this plays a
role too?

I only checked rsync1 last time, maybe rsync2 isn't as equal as I
thought.

Thanks,
Fabian

--
Fabian Groffen
Gentoo on a different level

signature.asc (201 bytes) Download Attachment