hi,
these two patches fix (I believe) the races recently reported
by Bodo Stroesser.
Thanks,
NeilBrown
---
NeilBrown (2):
sunrpc/cache: remove races with queuing an upcall.
sunrpc/cache: use cache_fresh_unlocked consistently and correctly.
net/sunrpc/cache.c | 36 +++++++++++++++++++++---------------
1 file changed, 21 insertions(+), 15 deletions(-)
--
Signature
We currently queue an upcall after setting CACHE_PENDING,
and dequeue after clearing CACHE_PENDING.
So a request should only be present when CACHE_PENDING is set.
However we don't combine the test and the enqueue/dequeue in
a protected region, so it is possible (if unlikely) for a race
to result in a request being queued without CACHE_PENDING set,
or a request to be absent despite CACHE_PENDING.
So: include a test for CACHE_PENDING inside the regions of
enqueue and dequeue where queue_lock is held, and abort
the operation if the value is not as expected.
With this, it perfectly safe and correct to:
- call cache_dequeue() if and only if we have just
cleared CACHE_PENDING
- call sunrpc_cache_pipe_upcall() (via cache_make_upcall)
if and only if we have just set CACHE_PENDING.
Reported-by: Bodo Stroesser <[email protected]>
Signed-off-by: NeilBrown <[email protected]>
---
net/sunrpc/cache.c | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/net/sunrpc/cache.c b/net/sunrpc/cache.c
index 9afa439..b48c8ef 100644
--- a/net/sunrpc/cache.c
+++ b/net/sunrpc/cache.c
@@ -1022,6 +1022,9 @@ static void cache_dequeue(struct cache_detail *detail, struct cache_head *ch)
struct cache_request *cr = container_of(cq, struct cache_request, q);
if (cr->item != ch)
continue;
+ if (test_bit(CACHE_PENDING, &ch->flags))
+ /* Lost a race and it is pending again */
+ break;
if (cr->readers != 0)
continue;
list_del(&cr->q.list);
@@ -1151,6 +1154,7 @@ int sunrpc_cache_pipe_upcall(struct cache_detail *detail, struct cache_head *h,
struct cache_request *crq;
char *bp;
int len;
+ int ret = 0;
if (!cache_listeners_exist(detail)) {
warn_no_listener(detail);
@@ -1182,10 +1186,18 @@ int sunrpc_cache_pipe_upcall(struct cache_detail *detail, struct cache_head *h,
crq->len = PAGE_SIZE - len;
crq->readers = 0;
spin_lock(&queue_lock);
- list_add_tail(&crq->q.list, &detail->queue);
+ if (test_bit(CACHE_PENDING, &h->flags))
+ list_add_tail(&crq->q.list, &detail->queue);
+ else
+ /* Lost a race, no longer PENDING, so don't enqueue */
+ ret = -EAGAIN;
spin_unlock(&queue_lock);
wake_up(&queue_wait);
- return 0;
+ if (ret == -EAGAIN) {
+ kfree(buf);
+ kfree(crq);
+ }
+ return ret;
}
EXPORT_SYMBOL_GPL(sunrpc_cache_pipe_upcall);
On 26 Feb 2013 15:02:01 +0100 Bodo Stroesser <[email protected]>
wrote:
> On 26 Feb 2013 07:37:00 +0100 NeilBrown <[email protected]> wrote:
>
> > We currently queue an upcall after setting CACHE_PENDING, and dequeue after clearing CACHE_PENDING.
> > So a request should only be present when CACHE_PENDING is set.
> >
> > However we don't combine the test and the enqueue/dequeue in a protected region, so it is possible (if unlikely) for a race to result in a request being queued without CACHE_PENDING set, or a request to be absent despite CACHE_PENDING.
> >
> > So: include a test for CACHE_PENDING inside the regions of enqueue and dequeue where queue_lock is held, and abort the operation if the value is not as expected.
> >
> > With this, it perfectly safe and correct to:
> > - call cache_dequeue() if and only if we have just
> > cleared CACHE_PENDING
> > - call sunrpc_cache_pipe_upcall() (via cache_make_upcall)
> > if and only if we have just set CACHE_PENDING.
> >
> > Reported-by: Bodo Stroesser <[email protected]>
> > Signed-off-by: NeilBrown <[email protected]>
>
> Sorry, I don't agree with this patch, as it fixes the first scenario of my mail
> from 24 Feb 2013, but AFAICS changes the second one (which has been a minor
> point that didn't need fixing necessarily) to a memory leak.
>
> I'll try to expain my doubts:
>
> Again, assume two threads calling cache_check() concurrently for the same cache
> entry of a cache that has a reader.
> Both threads get result -EAGAIN from cache_is_valid(). The second thread at that
> moment is interrupted and suspended for a while.
> The first thread sets CACHE_PENDING and queues an upcall request and sleeps
> waiting for the reply.
> The reader reads the request and replies accordingly. In sunrpc_cache_update()
> the reader changes the entry to CACHE_VALID and calls cache_fresh_unlocked().
> In cache_fresh_unlocked() it resets CACHE_PENDING. At this moment it is
> interrupted and suspended.
> Now the second thread wakes up, sets CACHE_PENDING again and queues a new upcall
> request.
> The reader wakes up and sees, that CACHE_PENDING is set again and does not
> dequeue the old request. --> memory leak (?)
Yes, I think you are right.
>
> In my opinion, there are two possible fixes that could replace this patch:
>
> 1) Don't call cache_dequeue() from cache_check(). Trying to dequeue something
> even if we know, that we haven't queued, looks strange to me. (And yes, I
> understand the reason, why you don't like it, but nevertheless I consider
> this the best solution.)
The reason for calling cache_dequeue() is that someone else might have queued
something. We are the last to leave so we turn out the lights - doesn't
matter that we didn't turn them on.
So I think the correct fix to the leak is to remove the "return" near the end
of cache_dequeue().
i.e. whoever clears CACHE_PENDING must remove all entries from the queue. If
someone else sets CACHE_PENDING they might not succeed, but it doesn't matter
as then someone else will come along and remove all the entries.
> This one would fix my first scenariop only, but not the second.
>
> 2) I think, the starting point of all trouble is in cache_check().
> Currently, if a thread has set CACHE_PENDING, is works using a
> possibly no longer valid state of the cache entry (rv).
> AFAICS, it would fix most of the problems to re-check the
> cache entry's state between setting CACHE_PENDING and the upcall.
> The upcall should be done only, if still necessary.
> This one could be combined with a new bit in the entry's state, that is
> set if a valid entry is updated (that is: replaced). Checking this
> bit also immediately before cache_make_upcall() is called would
> also fix my second scenario fully in that it avoids unnecessary
> upcalls.
Repeating the tests after setting CACHE_PENDING wouldn't hurt, but in almost
all cases it wouldn't help either. The races that could result in a second
unnecessary up-call are extremely unlikely. So I think the best approach is
not trying to avoid them, but making sure that they don't cause any harm.
This is best done with safe programming practices, like the "last one out
turns out the lights" pattern.
The "return" which I suggest removing is really a premature optimisation
which should never have been included. Without it we should be completely
safe.
??
NeilBrown
cache_fresh_unlocked() is called when a cache entry
has been updated and ensures that if there were any
pending upcalls, they are cleared.
So every time we update a cache entry, we should call this,
and this should be the only way that we try to clear
pending calls (that sort of uniformity makes code sooo much
easier to read).
try_to_negate_entry() will (possibly) mark an entry as
negative. If it doesn't, it is because the entry already
is VALID.
So the entry will be valid on exit, so it is appropriate to
call cache_fresh_unlocked().
So tidy up try_to_negate_entry() to do that, and remove
partial open-coded cache_fresh_unlocked() from the one
call-site of try_to_negate_entry().
In the other branch of the 'switch(cache_make_upcall())',
we again have a partial open-coded version of cache_fresh_unlocked().
Replace that with a real call.
And again in cache_clean(), use a real call to cache_fresh_unlocked().
These call sites might previously have called
cache_revisit_request() if CACHE_PENDING wasn't set.
This is never necessary because cache_revisit_request() can
only do anything if the item is in the cache_defer_hash,
However any time that an item is added to the cache_defer_hash
(setup_deferral), the code immediately tests CACHE_PENDING,
and removes the entry again if it is clear. So all other
places we only need to 'cache_revisit_request' if we've
just cleared CACHE_PENDING.
Reported-by: Bodo Stroesser <[email protected]>
Signed-off-by: NeilBrown <[email protected]>
---
net/sunrpc/cache.c | 20 +++++++-------------
1 file changed, 7 insertions(+), 13 deletions(-)
diff --git a/net/sunrpc/cache.c b/net/sunrpc/cache.c
index b48c8ef..7ebee64 100644
--- a/net/sunrpc/cache.c
+++ b/net/sunrpc/cache.c
@@ -228,15 +228,14 @@ static int try_to_negate_entry(struct cache_detail *detail, struct cache_head *h
write_lock(&detail->hash_lock);
rv = cache_is_valid(detail, h);
- if (rv != -EAGAIN) {
- write_unlock(&detail->hash_lock);
- return rv;
+ if (rv == -EAGAIN) {
+ set_bit(CACHE_NEGATIVE, &h->flags);
+ cache_fresh_locked(h, seconds_since_boot()+CACHE_NEW_EXPIRY);
+ rv = -ENOENT;
}
- set_bit(CACHE_NEGATIVE, &h->flags);
- cache_fresh_locked(h, seconds_since_boot()+CACHE_NEW_EXPIRY);
write_unlock(&detail->hash_lock);
cache_fresh_unlocked(h, detail);
- return -ENOENT;
+ return rv;
}
/*
@@ -275,13 +274,10 @@ int cache_check(struct cache_detail *detail,
if (!test_and_set_bit(CACHE_PENDING, &h->flags)) {
switch (cache_make_upcall(detail, h)) {
case -EINVAL:
- clear_bit(CACHE_PENDING, &h->flags);
- cache_revisit_request(h);
rv = try_to_negate_entry(detail, h);
break;
case -EAGAIN:
- clear_bit(CACHE_PENDING, &h->flags);
- cache_revisit_request(h);
+ cache_fresh_unlocked(h, detail);
break;
}
}
@@ -457,9 +453,7 @@ static int cache_clean(void)
current_index ++;
spin_unlock(&cache_list_lock);
if (ch) {
- if (test_and_clear_bit(CACHE_PENDING, &ch->flags))
- cache_dequeue(current_detail, ch);
- cache_revisit_request(ch);
+ cache_fresh_unlocked(ch, d);
cache_put(ch, d);
}
} else
SGksCgpzb3JyeSBmb3IgbXkgbGF0ZSByZXBseS4gSSB3YXMgcXVpdGUgYnVzeSBhbmQgbmVl
ZGVkIHNvbWUgdGltZQp0byB0aGluayBhYm91dCB0aGF0IGNvbXBsaWNhdGVkIHN0dWZmIGJl
Zm9yZSB3cml0aW5nIGFuIGFuc3dlci4KCgpPbiAyOCBGZWIgMjAxMyAwMDoyNDowMCArMDEw
MCBOZWlsQnJvd24gPG5laWxiQHN1c2UuZGU+IHdyb3RlOgoKPiBPbiAyNiBGZWIgMjAxMyAx
NTowMjowMSArMDEwMCBCb2RvIFN0cm9lc3NlciA8YnN0cm9lc3NlckB0cy5mdWppdHN1LmNv
bT4KPiB3cm90ZToKPiAKPiA+IE9uIDI2IEZlYiAyMDEzIDA3OjM3OjAwICswMTAwIE5laWxC
cm93biA8bmVpbGJAc3VzZS5kZT4gd3JvdGU6Cj4gPiAKPiA+ID4gV2UgY3VycmVudGx5IHF1
ZXVlIGFuIHVwY2FsbCBhZnRlciBzZXR0aW5nIENBQ0hFX1BFTkRJTkcsIGFuZCBkZXF1ZXVl
IGFmdGVyIGNsZWFyaW5nIENBQ0hFX1BFTkRJTkcuCj4gPiA+IFNvIGEgcmVxdWVzdCBzaG91
bGQgb25seSBiZSBwcmVzZW50IHdoZW4gQ0FDSEVfUEVORElORyBpcyBzZXQuCj4gPiA+IAo+
ID4gPiBIb3dldmVyIHdlIGRvbid0IGNvbWJpbmUgdGhlIHRlc3QgYW5kIHRoZSBlbnF1ZXVl
L2RlcXVldWUgaW4gYSBwcm90ZWN0ZWQgcmVnaW9uLCBzbyBpdCBpcyBwb3NzaWJsZSAoaWYg
dW5saWtlbHkpIGZvciBhIHJhY2UgdG8gcmVzdWx0IGluIGEgcmVxdWVzdCBiZWluZyBxdWV1
ZWQgd2l0aG91dCBDQUNIRV9QRU5ESU5HIHNldCwgb3IgYSByZXF1ZXN0IHRvIGJlIGFic2Vu
dCBkZXNwaXRlIENBQ0hFX1BFTkRJTkcuCj4gPiA+IAo+ID4gPiBTbzogaW5jbHVkZSBhIHRl
c3QgZm9yIENBQ0hFX1BFTkRJTkcgaW5zaWRlIHRoZSByZWdpb25zIG9mIGVucXVldWUgYW5k
IGRlcXVldWUgd2hlcmUgcXVldWVfbG9jayBpcyBoZWxkLCBhbmQgYWJvcnQgdGhlIG9wZXJh
dGlvbiBpZiB0aGUgdmFsdWUgaXMgbm90IGFzIGV4cGVjdGVkLgo+ID4gPiAKPiA+ID4gV2l0
aCB0aGlzLCBpdCBwZXJmZWN0bHkgc2FmZSBhbmQgY29ycmVjdCB0bzoKPiA+ID4gIC0gY2Fs
bCBjYWNoZV9kZXF1ZXVlKCkgaWYgYW5kIG9ubHkgaWYgd2UgaGF2ZSBqdXN0Cj4gPiA+ICAg
IGNsZWFyZWQgQ0FDSEVfUEVORElORwo+ID4gPiAgLSBjYWxsIHN1bnJwY19jYWNoZV9waXBl
X3VwY2FsbCgpICh2aWEgY2FjaGVfbWFrZV91cGNhbGwpCj4gPiA+ICAgIGlmIGFuZCBvbmx5
IGlmIHdlIGhhdmUganVzdCBzZXQgQ0FDSEVfUEVORElORy4KPiA+ID4gCj4gPiA+IFJlcG9y
dGVkLWJ5OiBCb2RvIFN0cm9lc3NlciA8YnN0cm9lc3NlckB0cy5mdWppdHN1LmNvbT4KPiA+
ID4gU2lnbmVkLW9mZi1ieTogTmVpbEJyb3duIDxuZWlsYkBzdXNlLmRlPgo+ID4gCj4gPiBT
b3JyeSwgSSBkb24ndCBhZ3JlZSB3aXRoIHRoaXMgcGF0Y2gsIGFzIGl0IGZpeGVzIHRoZSBm
aXJzdCBzY2VuYXJpbyBvZiBteSBtYWlsCj4gPiBmcm9tIDI0IEZlYiAyMDEzLCBidXQgQUZB
SUNTIGNoYW5nZXMgdGhlIHNlY29uZCBvbmUgKHdoaWNoIGhhcyBiZWVuIGEgbWlub3IKPiA+
IHBvaW50IHRoYXQgZGlkbid0IG5lZWQgZml4aW5nIG5lY2Vzc2FyaWx5KSB0byBhIG1lbW9y
eSBsZWFrLgo+ID4gCj4gPiBJJ2xsIHRyeSB0byBleHBhaW4gbXkgZG91YnRzOgo+ID4gCj4g
PiBBZ2FpbiwgYXNzdW1lIHR3byB0aHJlYWRzIGNhbGxpbmcgY2FjaGVfY2hlY2soKSBjb25j
dXJyZW50bHkgZm9yIHRoZSBzYW1lIGNhY2hlCj4gPiBlbnRyeSBvZiBhIGNhY2hlIHRoYXQg
aGFzIGEgcmVhZGVyLgo+ID4gQm90aCB0aHJlYWRzIGdldCByZXN1bHQgLUVBR0FJTiBmcm9t
IGNhY2hlX2lzX3ZhbGlkKCkuIFRoZSBzZWNvbmQgdGhyZWFkIGF0IHRoYXQKPiA+IG1vbWVu
dCBpcyBpbnRlcnJ1cHRlZCBhbmQgc3VzcGVuZGVkIGZvciBhIHdoaWxlLgo+ID4gVGhlIGZp
cnN0IHRocmVhZCBzZXRzIENBQ0hFX1BFTkRJTkcgYW5kIHF1ZXVlcyBhbiB1cGNhbGwgcmVx
dWVzdCBhbmQgc2xlZXBzCj4gPiB3YWl0aW5nIGZvciB0aGUgcmVwbHkuCj4gPiBUaGUgcmVh
ZGVyIHJlYWRzIHRoZSByZXF1ZXN0IGFuZCByZXBsaWVzIGFjY29yZGluZ2x5LiBJbiBzdW5y
cGNfY2FjaGVfdXBkYXRlKCkKPiA+IHRoZSByZWFkZXIgY2hhbmdlcyB0aGUgZW50cnkgdG8g
Q0FDSEVfVkFMSUQgYW5kIGNhbGxzIGNhY2hlX2ZyZXNoX3VubG9ja2VkKCkuCj4gPiBJbiBj
YWNoZV9mcmVzaF91bmxvY2tlZCgpIGl0IHJlc2V0cyBDQUNIRV9QRU5ESU5HLiBBdCB0aGlz
IG1vbWVudCBpdCBpcwo+ID4gaW50ZXJydXB0ZWQgYW5kIHN1c3BlbmRlZC4KPiA+IE5vdyB0
aGUgc2Vjb25kIHRocmVhZCB3YWtlcyB1cCwgc2V0cyBDQUNIRV9QRU5ESU5HIGFnYWluIGFu
ZCBxdWV1ZXMgYSBuZXcgdXBjYWxsCj4gPiByZXF1ZXN0Lgo+ID4gVGhlIHJlYWRlciB3YWtl
cyB1cCBhbmQgc2VlcywgdGhhdCBDQUNIRV9QRU5ESU5HIGlzIHNldCBhZ2FpbiBhbmQgZG9l
cyBub3QKPiA+IGRlcXVldWUgdGhlIG9sZCByZXF1ZXN0LiAtLT4gbWVtb3J5IGxlYWsgKD8p
Cj4gCj4gWWVzLCBJIHRoaW5rIHlvdSBhcmUgcmlnaHQuCj4gCj4gPiAKPiA+IEluIG15IG9w
aW5pb24sIHRoZXJlIGFyZSB0d28gcG9zc2libGUgZml4ZXMgdGhhdCBjb3VsZCByZXBsYWNl
IHRoaXMgcGF0Y2g6Cj4gPiAKPiA+IDEpIERvbid0IGNhbGwgY2FjaGVfZGVxdWV1ZSgpIGZy
b20gY2FjaGVfY2hlY2soKS4gVHJ5aW5nIHRvIGRlcXVldWUgc29tZXRoaW5nCj4gPiAgICBl
dmVuIGlmIHdlIGtub3csIHRoYXQgd2UgaGF2ZW4ndCBxdWV1ZWQsIGxvb2tzIHN0cmFuZ2Ug
dG8gbWUuIChBbmQgeWVzLCBJCj4gPiAgICB1bmRlcnN0YW5kIHRoZSByZWFzb24sIHdoeSB5
b3UgZG9uJ3QgbGlrZSBpdCwgYnV0IG5ldmVydGhlbGVzcyBJIGNvbnNpZGVyCj4gPiAgICB0
aGlzIHRoZSBiZXN0IHNvbHV0aW9uLikKPiAKPiBUaGUgcmVhc29uIGZvciBjYWxsaW5nIGNh
Y2hlX2RlcXVldWUoKSBpcyB0aGF0IHNvbWVvbmUgZWxzZSBtaWdodCBoYXZlIHF1ZXVlZAo+
IHNvbWV0aGluZy4gIFdlIGFyZSB0aGUgbGFzdCB0byBsZWF2ZSBzbyB3ZSB0dXJuIG91dCB0
aGUgbGlnaHRzIC0gZG9lc24ndAo+IG1hdHRlciB0aGF0IHdlIGRpZG4ndCB0dXJuIHRoZW0g
b24uCj4gCj4gU28gSSB0aGluayB0aGUgY29ycmVjdCBmaXggdG8gdGhlIGxlYWsgaXMgdG8g
cmVtb3ZlIHRoZSAicmV0dXJuIiBuZWFyIHRoZSBlbmQKPiBvZiBjYWNoZV9kZXF1ZXVlKCku
Cj4gaS5lLiB3aG9ldmVyIGNsZWFycyBDQUNIRV9QRU5ESU5HIG11c3QgcmVtb3ZlIGFsbCBl
bnRyaWVzIGZyb20gdGhlIHF1ZXVlLiAgSWYKPiBzb21lb25lIGVsc2Ugc2V0cyBDQUNIRV9Q
RU5ESU5HIHRoZXkgbWlnaHQgbm90IHN1Y2NlZWQsIGJ1dCBpdCBkb2Vzbid0IG1hdHRlcgo+
IGFzIHRoZW4gc29tZW9uZSBlbHNlIHdpbGwgY29tZSBhbG9uZyBhbmQgcmVtb3ZlIGFsbCB0
aGUgZW50cmllcy4KCkkgdGhpbmssIHRoZXJlIGFyZSB0d28gcG9zc2libGUgbWV0aG9kcy4g
VGhlIGZpcnN0IC0gd2hpY2ggSSB0aGluayB3YXMgdGhlCm1ldGhvZCBjaG9zZW4gd2hlbiB0
aGUgY29kZSB3YXMgZGV2ZWxvcGVkIGluIHRoZSBwYXN0IC0gaXMgdG8gaGF2ZSBhIHN0cmlj
dApoYW5kc2hha2Ugb2YgZW5xdWV1ZWluZyBhIHJlcXVlc3QgYW5kIGRlcXVldWVpbmcgaXQg
bGF0ZXIuIEJ1dCBhcyB3ZSBrbm93LCBpdAp3YXNuJ3QgaW1wbGVtZW50ZWQgY29ycmVjdGx5
LiAoTXkgcGF0Y2hlcyB3ZXJlIHdyaXR0ZW4gdG8gZml4IHRoaXMuKQpUaGUgc2Vjb25kIG1l
dGhvZCBpcyB3aGF0IHlvdSBhcmUgcHJlZmVycmluZzogbWFrZSBzdXJlIHRoYXQgdGhlIGxh
c3Qgb25lCmNsZWFyaW5nIENBQ0hFX1BFTkRJTkcgZGVxdWV1ZXMgZXZlcnl0aGluZy4gU28g
eW91ciBwYXRjaGVzIGFyZSBzb21lIGtpbmQgb2YKYSByZWRlc2lnbiwgSSB0aGluay4gV2l0
aG91dCB0aGUgInJldHVybiIgaW4gY2FjaGVfZGVxdWV1ZSgpLCBBRkFJQ1MgeW91cgpwYXRj
aGVzIHNob3VsZCBiZSBmaW5lLgoKPiAKPiA+ICAgIFRoaXMgb25lIHdvdWxkIGZpeCBteSBm
aXJzdCBzY2VuYXJpb3Agb25seSwgYnV0IG5vdCB0aGUgc2Vjb25kLgo+ID4gCj4gPiAyKSBJ
IHRoaW5rLCB0aGUgc3RhcnRpbmcgcG9pbnQgb2YgYWxsIHRyb3VibGUgaXMgaW4gY2FjaGVf
Y2hlY2soKS4KPiA+ICAgIEN1cnJlbnRseSwgaWYgYSB0aHJlYWQgaGFzIHNldCBDQUNIRV9Q
RU5ESU5HLCBpcyB3b3JrcyB1c2luZyBhIAo+ID4gICAgcG9zc2libHkgbm8gbG9uZ2VyIHZh
bGlkIHN0YXRlIG9mIHRoZSBjYWNoZSBlbnRyeSAocnYpLgo+ID4gICAgQUZBSUNTLCBpdCB3
b3VsZCBmaXggbW9zdCBvZiB0aGUgcHJvYmxlbXMgdG8gcmUtY2hlY2sgdGhlCj4gPiAgICBj
YWNoZSBlbnRyeSdzIHN0YXRlIGJldHdlZW4gc2V0dGluZyBDQUNIRV9QRU5ESU5HIGFuZCB0
aGUgdXBjYWxsLgo+ID4gICAgVGhlIHVwY2FsbCBzaG91bGQgYmUgZG9uZSBvbmx5LCBpZiBz
dGlsbCBuZWNlc3NhcnkuCj4gPiAgICBUaGlzIG9uZSBjb3VsZCBiZSBjb21iaW5lZCB3aXRo
IGEgbmV3IGJpdCBpbiB0aGUgZW50cnkncyBzdGF0ZSwgdGhhdCBpcwo+ID4gICAgc2V0IGlm
IGEgdmFsaWQgZW50cnkgaXMgdXBkYXRlZCAodGhhdCBpczogcmVwbGFjZWQpLiBDaGVja2lu
ZyB0aGlzCj4gPiAgICBiaXQgYWxzbyBpbW1lZGlhdGVseSBiZWZvcmUgY2FjaGVfbWFrZV91
cGNhbGwoKSBpcyBjYWxsZWQgd291bGQKPiA+ICAgIGFsc28gZml4IG15IHNlY29uZCBzY2Vu
YXJpbyBmdWxseSBpbiB0aGF0IGl0IGF2b2lkcyB1bm5lY2Vzc2FyeQo+ID4gICAgdXBjYWxs
cy4KPiAKPiBSZXBlYXRpbmcgdGhlIHRlc3RzIGFmdGVyIHNldHRpbmcgQ0FDSEVfUEVORElO
RyB3b3VsZG4ndCBodXJ0LCBidXQgaW4gYWxtb3N0Cj4gYWxsIGNhc2VzIGl0IHdvdWxkbid0
IGhlbHAgZWl0aGVyLiAgVGhlIHJhY2VzIHRoYXQgY291bGQgcmVzdWx0IGluIGEgc2Vjb25k
Cj4gdW5uZWNlc3NhcnkgdXAtY2FsbCBhcmUgZXh0cmVtZWx5IHVubGlrZWx5LiAgU28gSSB0
aGluayB0aGUgYmVzdCBhcHByb2FjaCBpcwo+IG5vdCB0cnlpbmcgdG8gYXZvaWQgdGhlbSwg
YnV0IG1ha2luZyBzdXJlIHRoYXQgdGhleSBkb24ndCBjYXVzZSBhbnkgaGFybS4KPiBUaGlz
IGlzIGJlc3QgZG9uZSB3aXRoIHNhZmUgcHJvZ3JhbW1pbmcgcHJhY3RpY2VzLCBsaWtlIHRo
ZSAibGFzdCBvbmUgb3V0Cj4gdHVybnMgb3V0IHRoZSBsaWdodHMiIHBhdHRlcm4uICAKClRo
ZSByYWNlcyBJJ3ZlIGZvdW5kIHdlcmUgYSBjb25zZXF1ZW5jZSBvZiB0d28gdGhyZWFkcyBj
YWxsaW5nIGNhY2hlX2lzX3ZhbGlkKCkKY29uY3VycmVudGx5IGFuZCB0aHVzIGJvdGggdHJ5
aW5nIHRvIG1ha2UgYW4gdXBjYWxsLiBUaGUgZmlyc3QgdGhyZWFkIHNldHMKQ0FDSEVfUEVO
RElORyBhbmQgdHJpZXMgdGhlIHVwY2FsbC4gVGhlIHNlY29uZCB0aHJlYWQgY2FuIGRvIGl0
cyB1cGNhbGwgb25seSwKYWZ0ZXIgQ0FDSEVfUEVORElORyB3YXMgcmVzZXQgYWdhaW4gKHdo
aWNoIGNhbiBiZSBkb25lIGJ5IHRoZSBmaXJzdCB0aHJlYWQKaXRzZWxmIGlmIHRoZSB1cGNh
bGwgZmFpbGVkLCBvciBieSBhIHJlYWRlciBpZiB0aGUgdXBjYWxsIGlzIGFuc3dlcmVkKS4K
SW4gdGhpcyBjYXNlLCBhZnRlciB0aGUgc2Vjb25kIHRocmVhZCBoYXMgc2V0IENBQ0hFX1BF
TkRJTkcgaXRzZWxmLCB0aGUgc3RhdGUKb2YgdGhlIGNhY2hlIGVudHJ5IHdpbGwgaGF2ZSBj
aGFuZ2VkLiBTbywgcmVjaGVja2luZyB0aGUgc3RhdGUgY2FuIGF2b2lkIHRoZQpzZWNvbmQg
dXBjYWxsIGFuZCB0aHVzIGFsc28gYXZvaWQgdGhlIHBvc3NpYmlsaXR5IG9mIHJhY2VzLgoK
PiBUaGUgInJldHVybiIgd2hpY2ggSSBzdWdnZXN0IHJlbW92aW5nIGlzIHJlYWxseSBhIHBy
ZW1hdHVyZSBvcHRpbWlzYXRpb24KPiB3aGljaCBzaG91bGQgbmV2ZXIgaGF2ZSBiZWVuIGlu
Y2x1ZGVkLiAgV2l0aG91dCBpdCB3ZSBzaG91bGQgYmUgY29tcGxldGVseQo+IHNhZmUuCgpG
b3IgdGhlIG9sZCBtZXRob2Qgb2YgYW4gZW5xdWV1ZWluZyAvIGRlcXVldWVpbmcgaGFuZHNo
YWtlIEkgdGhpbmsgaXQgd2FzCmZpbmUuCgo+IAo+ID8/CgpBRkFJQ1MsIHlvdXIgbmV3ZXN0
IHBhdGNoZXMgdGhhdCByZW1vdmUgdGhlICJyZXR1cm4iIHNob3VsZCBiZSBmaW5lLgpVbmZv
cnR1bmF0ZWx5LCBJIGNhbid0IHRlc3QgaXQsIGFzIG91ciBzZXR1cCBpcyBiYXNlZCBvbiBh
IFNMRVMxMSBTUDEsCndoaWNoIGNhbid0IGJlIGNoYW5nZWQgd2l0aG91dCBjaGFuZ2luZyBh
IGxvdCBvZiBvdGhlciBTVyBhbHNvLgpUaHVzLCBJJ2xsIHRyeSB0byBiYWNrcG9ydCB0aGUg
cGF0Y2hlcyBhbmQgZG8gYSB0ZXN0LgoKQm9kbwoKPiAKPiBOZWlsQnJvd24KPiAK