Received: by 2002:ab2:69cc:0:b0:1f4:be93:e15a with SMTP id n12csp1665893lqp; Mon, 15 Apr 2024 13:10:21 -0700 (PDT) X-Forwarded-Encrypted: i=2; AJvYcCVul4GtV6xY4tCtKb0CH+U/lLfcgA0MimcTpSCOEc6UdoTFFL6ktDdNovUJO/r20P7DoKKmmow0cfP8Rfy+0Uw/iS3t3lLWoE20mxEGDw== X-Google-Smtp-Source: AGHT+IEOkFXD5i71k2rL8e7dG1irj7fYdfx8eb/jRgakjllwaxxXwA7Gj8w/tb3dgeZDzo9oEgqa X-Received: by 2002:a17:907:60ce:b0:a55:3463:ce54 with SMTP id hv14-20020a17090760ce00b00a553463ce54mr36251ejc.8.1713211821060; Mon, 15 Apr 2024 13:10:21 -0700 (PDT) Return-Path: Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id bv7-20020a170907934700b00a51c35fe149si4990569ejc.850.2024.04.15.13.10.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Apr 2024 13:10:21 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel+bounces-145862-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@broadcom.com header.s=google header.b="Vv5bqas/"; arc=fail (body hash mismatch); spf=pass (google.com: domain of linux-kernel+bounces-145862-linux.lists.archive=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-145862-linux.lists.archive=gmail.com@vger.kernel.org"; dmarc=fail (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=broadcom.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 674751F2268E for ; Mon, 15 Apr 2024 20:10:20 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7977B15666F; Mon, 15 Apr 2024 20:10:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b="Vv5bqas/" Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF75C156653 for ; Mon, 15 Apr 2024 20:10:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.170 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713211810; cv=none; b=FM0uPEjYXT0IVgyO3iwrL321Iln1X6yCYTfbNheHGJvScgVKoA4LR4JqZrMLDfoiG3KSSYQ7LYhn+2evmgkOm/EKsIknl3MsSgU89scn6P5x/KkB8Ydsmrg6nLXC7qX+JTUJtp7qKD92h1+bKQy23FBtqxD1HnUoL53QDhD0smg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713211810; c=relaxed/simple; bh=5d4gG6KcSrnltQEURVwc1z7O22ov+6zFqp+A9lGwXng=; h=MIME-Version:References:In-Reply-To:From:Date:Message-ID:Subject: To:Cc:Content-Type; b=DcghSPF3jbV2LVq2OHDdmQSBau3gYLnNjkOt4HSqICM5sYrqzmKkzUoEUwDxJL86nQr/eXEzr3shXQk9RvuOXQQ61XRcjgpU8ugPLwusCv9ag8gc8OQt3Va5vxXfE3+knfinpHn+PLo85eM2ca3MX658jpMq50+frzF1CeoPLY0= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com; spf=fail smtp.mailfrom=broadcom.com; dkim=pass (1024-bit key) header.d=broadcom.com header.i=@broadcom.com header.b=Vv5bqas/; arc=none smtp.client-ip=209.85.219.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=broadcom.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=broadcom.com Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-dcbd1d4904dso3557705276.3 for ; Mon, 15 Apr 2024 13:10:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1713211807; x=1713816607; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=CfN9Wb1/+M3BuV12oSEhCVjog+nl1Y/F6VdK3xFRU9M=; b=Vv5bqas/a1ALJdc/v4JOsYongB8S7eS3R2tQuc5INacgc5w2h6DcppBfem4ddjNCHE r3GeL0cBKeRM4wJGr7pdj5FAyhDg2QWCEHB1JvrODF5WfjA7Q9MKLpDVy4NlzLPT+x3h Pok8PUjC82Nr0p9GwcUFye+feE0ousiZ6x5Tw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713211807; x=1713816607; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=CfN9Wb1/+M3BuV12oSEhCVjog+nl1Y/F6VdK3xFRU9M=; b=oWG8EuCOeWHHfgVB362NsTMaTgmdEQpIZFJ52wBm9GBJUVHOkAhOjx/NJmVCKTFZFp esjDXEkVh9/FTA6b3hBuEn/bbOpq9ZEGfnLHTtEiBQHPARoXk6H19mtUFeelDe42qtuA ej9SgycuIthQU83ykJ1S3WHM47wIt4i80bGTUK7Ok8+BKpLrjiDHbjIVG4jRbv75n75U qSjHE4NfoLQTEKDAeFSj8IVSw27enQaenV81UFZ53euawLbdtrTGDQ9pUCy1rTXErlPt xqxg5hcg3VmC036xS1PjbsySSZAMLs+3+8guOKWf1s9LbUyptmqQoxJ/LKHAQixESPqu q8ig== X-Forwarded-Encrypted: i=1; AJvYcCU5YD4KyQdzB4p2Qy0Y37QwP99+f4UTYFsC6EHkN0n73nlhA+lKiHALPMQNQ0noLb20BbmFwH0rYw9eNYA947hpY0i8grdoLvGQimEn X-Gm-Message-State: AOJu0Yz6YUTi40LU7IvEGfeQHVsDCZFVXV/9XwfCJkGkiDv4izUkX9+V zfHcURR/YGqxPUDC3xqALx59lCEHjKTmoSVlQXVQrJdUhYdz6hqXyymE7nKBfpQgx8AJj0qM/gh Ku9YGSLAFPACStpn0zf1JYS8F20NfaeUDxoDuXVHHaiCnm4hMrFC2euMwCnRjzBVgSV8znalm3f IK/1Oo/M31ZIhEFrJFEeGDuOxP+g== X-Received: by 2002:a05:6902:2408:b0:dc7:8c3a:4e42 with SMTP id dr8-20020a056902240800b00dc78c3a4e42mr11888592ybb.30.1713211806317; Mon, 15 Apr 2024 13:10:06 -0700 (PDT) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 References: <20240321174423.00007e0d@Huawei.com> <66109191f3d6d_2583ad29468@dwillia2-xfh.jf.intel.com.notmuch> <20240410124517.000075f2@Huawei.com> In-Reply-To: <20240410124517.000075f2@Huawei.com> From: Sreenivas Bagalkote Date: Mon, 15 Apr 2024 14:09:38 -0600 Message-ID: Subject: Re: RFC: Restricting userspace interfaces for CXL fabric management To: Jonathan Cameron , Dan Williams Cc: linux-cxl@vger.kernel.org, Brett Henning , Harold Johnson , Sumanesh Samanta , linux-kernel@vger.kernel.org, Davidlohr Bueso , Dave Jiang , Alison Schofield , Vishal Verma , Ira Weiny , linuxarm@huawei.com, linux-api@vger.kernel.org, Lorenzo Pieralisi , "Natu, Mahesh" Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-256; boundary="0000000000002fd1250616283586" --0000000000002fd1250616283586 Content-Type: multipart/alternative; boundary="000000000000243380061628359f" --000000000000243380061628359f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, >> We need guidance from the community. >> 1. Datacenter customers must be able to manage PCIe switches in-band.>> = 2. Management of switches includes getting health, performance, and error t= elemetry.>> 3. These telemetry functions are not yet part of the CXL standa= rd>> 4. We built the CCI mailboxes into our PCIe switches per CXL spec and = developed our management scheme around them.>> >> If the Linux community do= es not allow a CXL spec-compliant switch to be>> managed via the CXL spec-d= efined CCI mailbox, then please guide us on>> the right approach. Please te= ll us how you propose we manage our switches>> in-band. I am still looking for your guidance. We need to be able to manage our switch via the CCI mailbox. We need to use vendor-defined commands per CXL spec. You talked about whitelisting commands (allow-list) which we agreed to. Would you please confirm that you will allow the vendor-defined allow-list of commands? Thank you Sreeni On Wed, Apr 10, 2024 at 5:45=E2=80=AFAM Jonathan Cameron < Jonathan.Cameron@huawei.com> wrote: > On Fri, 5 Apr 2024 17:04:34 -0700 > Dan Williams wrote: > > Hi Dan, > > > Jonathan Cameron wrote: > > > Hi All, > > > > > > This is has come up in a number of discussions both on list and in > private, > > > so I wanted to lay out a potential set of rules when deciding whether > or not > > > to provide a user space interface for a particular feature of CXL > Fabric > > > Management. The intent is to drive discussion, not to simply tell > people > > > a set of rules. I've brought this to the public lists as it's a Linu= x > kernel > > > policy discussion, not a standards one. > > > > > > Whilst I'm writing the RFC this my attempt to summarize a possible > > > position rather than necessarily being my personal view. > > > > > > It's a straw man - shoot at it! > > > > > > Not everyone in this discussion is familiar with relevant kernel or > CXL concepts > > > so I've provided more info than I normally would. > > > > Thanks for writing this up Jonathan! > > > > [..] > > > 2) Unfiltered userspace use of mailbox for Fabric Management - BMC > kernels > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > (This would just be a kernel option that we'd advise normal server > > > distributions not to turn on. Would be enabled by openBMC etc) > > > > > > This is fine - there is some work to do, but the switch-cci PCI drive= r > > > will hopefully be ready for upstream merge soon. There is no filterin= g > of > > > accesses. Think of this as similar to all the damage you can do via > > > MCTP from a BMC. Similarly it is likely that much of the complexity > > > of the actual commands will be left to user space tooling: > > > https://gitlab.com/jic23/cxl-fmapi-tests has some test examples. > > > > > > Whether Kconfig help text is strong enough to ensure this only gets > > > enabled for BMC targeted distros is an open question we can address > > > alongside an updated patch set. > > > > It is not clear to me that this material makes sense to house in > > drivers/ vs tools/ or even out-of-tree just for maintenance burden > > relief of keeping the universes separated. What does the Linux kernel > > project get out of carrying this in mainline alongside the inband code? > > I'm not sure what you mean by in band. Aim here was to discuss > in-band drivers for switch CCI etc. Same reason from a kernel point of > view for why we include embedded drivers. I'll interpret in band > as host driven and not inband as FM-API stuff. > > > I do think the mailbox refactoring to support non-CXL use cases is > > interesting, but only so far as refactoring is consumed for inband use > > cases like RAS API. > > If I read this right, I disagree with the 'only so far' bit. > > In all substantial ways we should support BMC use case of the Linux Kerne= l > at a similar level to how we support forms of Linux Distros. It may > not be our target market as developers for particular parts of our > companies, > but we should not block those who want to support it. > > We should support them in drivers/ - maybe with example userspace code > in tools. Linux distros on BMCs is a big market, there are a number > of different distros using (and in some cases contributing to) the > upstream kernel. Not everyone is using openBMC so there is not one > common place where downstream patches could be carried. > From a personal point of view, I like that for the same reasons that > I like there being multiple Linux sever focused distros. It's a sign > of a healthy ecosystem to have diverse options taking the mainline > kernel as their starting point. > > BMCs are just another embedded market, and like other embedded markets > we want to encourage upstream first etc. > openBMC has a policy on this: > https://github.com/openbmc/docs/blob/master/kernel-development.md > "The OpenBMC project maintains a kernel tree for use by the project. > The tree's general development policy is that code must be upstream > first." There are paths to bypass that for openBMC so it's a little > more relaxed than some enterprise distros (today, their policies used > to look very similar to this) but we should not be telling > them they need to carry support downstream. If we are > going to tell them that, we need to be able to point at a major > sticking point for maintenance burden. So far I don't see the > additional complexity as remotely close reaching that bar. > > So I think we do want switch-cci support and for that matter the equivale= nt > for MHDs in the upstream kernel. > > One place I think there is some wiggle room is the taint on use of raw > commands. Leaving removal of that for BMC kernels as a patch they need > to carry downstream doesn't seem too burdensome. I'm sure they'll push > back if it is a problem for them! So I think we can kick that question > into the future. > > Addressing maintenance burden, there is a question of where we split > the stack. Ignore MHDs for now (I won't go into why in this forum...) > > The current proposal is (simplified to ignore some sharing in lookup code > etc > that I can rip out if we think it might be a long term problem) > > _____________ _____________________ > | | | | > | Switch CCI | | Type 3 Driver stack| > |_____________| |_____________________| > |___________________________| Whatever GPU etc > _______|_______ _______|______ > | | | | > | CXL MBOX | | RAS API etc | > |_______________| |______________| > |_____________________________| > | > _________|______ > | | > | MMPT mbox | > |________________| > > Switch CCI Driver: PCI driver doing everything beyond the CXL mbox > specific bit. > Type 3 Stack: All the normal stack just with the CXL Mailbox specific > stuff factored > out. Note we can move different amounts of shared logic in > here, but > in essence it deals with the extra layer on top of the raw > MMPT mbox. > MMPT Mbox: Mailbox as per the PCI spec. > RAS API: Shared RAS API specific infrastructure used by other drivers. > > If we see a significant maintenance burden, maybe we duplicate the CXL > specific > MBOX layer - I can see advantages in that as there is some stuff not > relevant > to the Switch CCI. There will be some duplication of logic however such > as background command support (which is CXL only IIUC) We can even use > a difference IOCTL number so the two can diverge if needed in the long ru= n. > > e.g. If it makes it easier to get upstream, we can merrily duplicated cod= e > so that only the bit common with RAS API etc is shared (assuming the > actually end up with MMPT, not the CXL mailbox which is what their curren= t > publicly available spec talks about and I assume is a pref MMPT left over= ?) > > _____________ _____________________ > | | | | > | Switch CCI | | Type 3 Driver stack| > |_____________| |_____________________| > | | Whatever GPU etc > _______|_______ _______|_______ ______|_______ > | | | | | | > | CXL MBOX | | CXL MBOX | | RAS API etc | > |_______________| |_______________| |______________| > |_____________________________|____________________| > | > ________|______ > | | > | MMPT mbox | > |_______________| > > > > > (On to the one that the "debate" is about) > > > > > > 3) Unfiltered user space use of mailbox for Fabric Management - Distr= o > kernels > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > > > (General purpose Linux Server Distro (Redhat, Suse etc)) > > > > > > This is equivalent of RAW command support on CXL Type 3 memory device= s. > > > You can enable those in a distro kernel build despite the scary confi= g > > > help text, but if you use it the kernel is tainted. The result > > > of the taint is to add a flag to bug reports and print a big message > to say > > > that you've used a feature that might result in you shooting yourself > > > in the foot. > > > > > > The taint is there because software is not at first written to deal > with > > > everything that can happen smoothly (e.g. surprise removal) It's hard > > > to survive some of these events, so is never on the initial feature > list > > > for any bus, so this flag is just to indicate we have entered a world > > > where almost all bets are off wrt to stability. We might not know wh= at > > > a command does so we can't assess the impact (and no one trusts vendo= r > > > commands to report affects right in the Command Effects Log - which > > > in theory tells you if a command can result problems). > > > > That is a secondary reason that the taint is there. Yes, it helps > > upstream not waste their time on bug reports from proprietary use cases= , > > but the effect of that is to make "raw" command mode unattractive for > > deploying solutions at scale. It clarifies that this interface is a > > debug-tool that enterprise environment need not worry about. > > > > The more salient reason for the taint, speaking only for myself as a > > Linux kernel community member not for $employer, is to encourage open > > collaboration. Take firmware-update for example that is a standard > > command with known side effects that is inaccessible via the ioctl() > > path. It is placed behind an ABI that is easier to maintain and reason > > about. Everyone has the firmware update tool if they have the 'cat' > > command. Distros appreciate the fact that they do not need ship yet > > another vendor device-update tool, vendors get free tooling and end > > users also appreciate one flow for all devices. > > > > As I alluded here [1], I am not against innovation outside of the > > specification, but it needs to be open, and it needs to plausibly becom= e > > if not a de jure standard at least a de facto standard. > > > > [1]: > https://lore.kernel.org/all/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8ZkcmHnP= p3B0cw@mail.gmail.com/ > > Agree with all this. > > > > > > A concern was raised about GAE/FAST/LDST tables for CXL Fabrics > > > (a r3.1 feature) but, as I understand it, these are intended for a > > > host to configure and should not have side effects on other hosts? > > > My working assumption is that the kernel driver stack will handle > > > these (once we catch up with the current feature backlog!) Currently > > > we have no visibility of what the OS driver stack for a fabrics will > > > actually look like - the spec is just the starting point for that. > > > (patches welcome ;) > > > > > > The various CXL upstream developers and maintainers may have > > > differing views of course, but my current understanding is we want > > > to support 1 and 2, but are very resistant to 3! > > > > 1, yes, 2, need to see the patches, and agree on 3. > > If we end up with top architecture of the diagrams above, 2 will look > pretty > similar to last version of the switch-cci patches. So raw commands only = + > taint. > Factoring out MMPT is another layer that doesn't make that much differenc= e > in > practice to this discussion. Good to have, but the reuse here would be on= e > layer > above that. > > Or we just say go for second proposed architecture and 0 impact on the > CXL specific code, just reuse of the MMPT layer. I'd imagine people will > get > grumpy on code duplication (and we'll spend years rejecting patch sets th= at > try to share the cdoe) but there should be no maintenance burden as > a result. > > > > > > General Notes > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > > > One side aspect of why we really don't like unfiltered userspace > access to any > > > of these devices is that people start building non standard hacks in > and we > > > lose the ecosystem advantages. Forcing a considered discussion + > patches > > > to let a particular command be supported, drives standardization. > > > > Like I said above, I think this is not a side aspect. It is fundamental > > to the viability Linux as a project. This project only works because > > organizations with competing goals realize they need some common > > infrastructure and that there is little to be gained by competing on th= e > > commons. > > > > > > https://lore.kernel.org/linux-cxl/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8Z= kcmHnPp3B0cw@mail.gmail.com/ > > > provides some history on vendor specific extensions and why in genera= l > we > > > won't support them upstream. > > > > Oh, you linked my writeup... I will leave the commentary I added here i= n > case > > restating it helps. > > > > > To address another question raised in an earlier discussion: > > > Putting these Fabric Management interfaces behind guard rails of some > type > > > (e.g. CONFIG_IM_A_BMC_AND_CAN_MAKE_A_MESS) does not encourage the ris= k > > > of non standard interfaces, because we will be even less likely to > accept > > > those upstream! > > > > > > If anyone needs more details on any aspect of this please ask. > > > There are a lot of things involved and I've only tried to give a fair= ly > > > minimal illustration to drive the discussion. I may well have missed > > > something crucial. > > > > You captured it well, and this is open source so I may have missed > > something crucial as well. > > > > Thanks for detailed reply! > > Jonathan > > > --=20 This electronic communication and the information and any files transmitted= =20 with it, or attached to it, are confidential and are intended solely for=20 the use of the individual or entity to whom it is addressed and may contain= =20 information that is confidential, legally privileged, protected by privacy= =20 laws, or otherwise restricted from disclosure to anyone else. If you are=20 not the intended recipient or the person responsible for delivering the=20 e-mail to the intended recipient, you are hereby notified that any use,=20 copying, distributing, dissemination, forwarding, printing, or copying of= =20 this e-mail is strictly prohibited. If you received this e-mail in error,= =20 please return the e-mail to the sender, delete it from your computer, and= =20 destroy any printed copy of it. --000000000000243380061628359f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,
<= span style=3D"color:rgb(0,0,51)">
>> We need guidance = from the community.
>>=C2=A01. Datacenter customers must be able to manage PCIe sw= itches in-band. >>=C2=A02. Management of switches includes getting health, performance, and error = telemetry. >>=C2=A03. These telemetry functions are not yet part of the CXL standard >>=C2=A04. We built the CCI mailboxes into our PCIe switches per CXL spec and deve= loped our management scheme around them. >>=C2=A0 >>=C2=A0If the Linux community does not allow a CXL spec-compliant switch to be >>=C2=A0managed via the CXL spec-defined CCI mailbox, then please guide us on >>=C2=A0the right approach. Please tell us how you propose we manage our switches >>=C2=A0in-band.
I am still looking for your guidance. We need to = be able to manage our switch via the CCI mailbox. We need to use vendor-def= ined commands per CXL spec.

You t= alked about whitelisting commands (allow-list) which we agreed to. Would yo= u please confirm that you will allow the vendor-defined allow-list of comma= nds?

Thank you
Sr= eeni

On Wed, Apr 10, 2024 at 5:45=E2=80=AFAM Jonat= han Cameron <Jonathan.Cam= eron@huawei.com> wrote:
On Fri, 5 Apr 2024 17:04:34 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

Hi Dan,

> Jonathan Cameron wrote:
> > Hi All,
> >
> > This is has come up in a number of discussions both on list and i= n private,
> > so I wanted to lay out a potential set of rules when deciding whe= ther or not
> > to provide a user space interface for a particular feature of CXL= Fabric
> > Management.=C2=A0 The intent is to drive discussion, not to simpl= y tell people
> > a set of rules.=C2=A0 I've brought this to the public lists a= s it's a Linux kernel
> > policy discussion, not a standards one.
> >
> > Whilst I'm writing the RFC this my attempt to summarize a pos= sible
> > position rather than necessarily being my personal view.
> >
> > It's a straw man - shoot at it!
> >
> > Not everyone in this discussion is familiar with relevant kernel = or CXL concepts
> > so I've provided more info than I normally would.=C2=A0
>
> Thanks for writing this up Jonathan!
>
> [..]
> > 2) Unfiltered userspace use of mailbox for Fabric Management - BM= C kernels
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D
> >
> > (This would just be a kernel option that we'd advise normal s= erver
> > distributions not to turn on. Would be enabled by openBMC etc) > >
> > This is fine - there is some work to do, but the switch-cci PCI d= river
> > will hopefully be ready for upstream merge soon. There is no filt= ering of
> > accesses. Think of this as similar to all the damage you can do v= ia
> > MCTP from a BMC. Similarly it is likely that much of the complexi= ty
> > of the actual commands will be left to user space tooling:
> > https://gitlab.com/jic23/cxl-fmapi-tests has s= ome test examples.
> >
> > Whether Kconfig help text is strong enough to ensure this only ge= ts
> > enabled for BMC targeted distros is an open question we can addre= ss
> > alongside an updated patch set.=C2=A0
>
> It is not clear to me that this material makes sense to house in
> drivers/ vs tools/ or even out-of-tree just for maintenance burden
> relief of keeping the universes separated. What does the Linux kernel<= br> > project get out of carrying this in mainline alongside the inband code= ?

I'm not sure what you mean by in band.=C2=A0 Aim here was to discuss in-band drivers for switch CCI etc. Same reason from a kernel point of
view for why we include embedded drivers.=C2=A0 I'll interpret in band<= br> as host driven and not inband as FM-API stuff.

> I do think the mailbox refactoring to support non-CXL use cases is
> interesting, but only so far as refactoring is consumed for inband use=
> cases like RAS API.

If I read this right, I disagree with the 'only so far' bit.

In all substantial ways we should support BMC use case of the Linux Kernel<= br> at a similar level to how we support forms of Linux Distros.=C2=A0 It may not be our target market as developers for particular parts of our companie= s,
but we should not block those who want to support it.

We should support them in drivers/ - maybe with example userspace code
in tools.=C2=A0 Linux distros on BMCs is a big market, there are a number of different distros using (and in some cases contributing to) the
upstream kernel. Not everyone is using openBMC so there is not one
common place where downstream patches could be carried.
From a personal point of view, I like that for the same reasons that
I like there being multiple Linux sever focused distros. It's a sign of a healthy ecosystem to have diverse options taking the mainline
kernel as their starting point.

BMCs are just another embedded market, and like other embedded markets
we want to encourage upstream first etc.
openBMC has a policy on this:
https://github.com/openbmc/docs/blo= b/master/kernel-development.md
"The OpenBMC project maintains a kernel tree for use by the project. The tree's general development policy is that code must be upstream
first." There are paths to bypass that for openBMC so it's a littl= e
more relaxed than some enterprise distros (today, their policies used
to look very similar to this) but we should not be telling
them they need to carry support downstream.=C2=A0 If we are
going to tell them that, we need to be able to point at a major
sticking point for maintenance burden.=C2=A0 So far I don't see the
additional complexity as remotely close reaching that bar.

So I think we do want switch-cci support and for that matter the equivalent=
for MHDs in the upstream kernel.

One place I think there is some wiggle room is the taint on use of raw
commands.=C2=A0 Leaving removal of that for BMC kernels as a patch they nee= d
to carry downstream doesn't seem too burdensome. I'm sure they'= ll push
back if it is a problem for them!=C2=A0 So I think we can kick that questio= n
into the future.

Addressing maintenance burden, there is a question of where we split
the stack.=C2=A0 Ignore MHDs for now (I won't go into why in this forum= ..)

The current proposal is (simplified to ignore some sharing in lookup code e= tc
that I can rip out if we think it might be a long term problem)

=C2=A0 =C2=A0 =C2=A0_____________=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _______= ______________
=C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 | Switch CCI=C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 Type = 3 Driver stack|
=C2=A0 =C2=A0 |_____________|=C2=A0 =C2=A0 =C2=A0 =C2=A0 |_________________= ____|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|___________________________|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Whatever GPU etc
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _______|____= ___=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0___= ____|______
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 CXL M= BOX=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0| RAS API etc=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|____________= ___|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|________= ______|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|_____________________________|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _________|______
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0MMPT mbox=C2=A0 = =C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|________________|

Switch CCI Driver: PCI driver doing everything beyond the CXL mbox specific= bit.
Type 3 Stack: All the normal stack just with the CXL Mailbox specific stuff= factored
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 out. Note we can move diff= erent amounts of shared logic in here, but
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 in essence it deals with t= he extra layer on top of the raw MMPT mbox.
MMPT Mbox: Mailbox as per the PCI spec.
RAS API:=C2=A0 =C2=A0Shared RAS API specific infrastructure used by other d= rivers.

If we see a significant maintenance burden, maybe we duplicate the CXL spec= ific
MBOX layer - I can see advantages in that as there is some stuff not releva= nt
to the Switch CCI.=C2=A0 There will be some duplication of logic however su= ch
as background command support (which is CXL only IIUC)=C2=A0 We can even us= e
a difference IOCTL number so the two can diverge if needed in the long run.=

e.g. If it makes it easier to get upstream, we can merrily duplicated code<= br> so that only the bit common with RAS API etc is shared (assuming the
actually end up with MMPT, not the CXL mailbox which is what their current<= br> publicly available spec talks about and I assume is a pref MMPT left over?)=

=C2=A0 =C2=A0 =C2=A0_____________=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 _______= ______________
=C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2= =A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 | Switch CCI=C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 Type = 3 Driver stack|
=C2=A0 =C2=A0 |_____________|=C2=A0 =C2=A0 =C2=A0 =C2=A0 |_________________= ____|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Whatever GPU etc
=C2=A0 =C2=A0 _______|_______=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0_______|_______=C2=A0 =C2=A0 =C2=A0 =C2=A0 ______|_______
=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0|=C2=A0 CXL MBOX=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0|=C2=A0 CXL MBOX=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 = | RAS API etc=C2=A0 |
=C2=A0 =C2=A0|_______________|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|___= ____________|=C2=A0 =C2=A0 =C2=A0 |______________|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|_____________________________|___= _________________|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|<= br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0________|______
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0MMPT mbox=C2=A0 =C2=A0= |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |_______________|


> > (On to the one that the "debate" is about)
> >
> > 3) Unfiltered user space use of mailbox for Fabric Management - D= istro kernels
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D
> > (General purpose Linux Server Distro (Redhat, Suse etc))
> >
> > This is equivalent of RAW command support on CXL Type 3 memory de= vices.
> > You can enable those in a distro kernel build despite the scary c= onfig
> > help text, but if you use it the kernel is tainted. The result > > of the taint is to add a flag to bug reports and print a big mess= age to say
> > that you've used a feature that might result in you shooting = yourself
> > in the foot.
> >
> > The taint is there because software is not at first written to de= al with
> > everything that can happen smoothly (e.g. surprise removal) It= 9;s hard
> > to survive some of these events, so is never on the initial featu= re list
> > for any bus, so this flag is just to indicate we have entered a w= orld
> > where almost all bets are off wrt to stability.=C2=A0 We might no= t know what
> > a command does so we can't assess the impact (and no one trus= ts vendor
> > commands to report affects right in the Command Effects Log - whi= ch
> > in theory tells you if a command can result problems).=C2=A0
>
> That is a secondary reason that the taint is there. Yes, it helps
> upstream not waste their time on bug reports from proprietary use case= s,
> but the effect of that is to make "raw" command mode unattra= ctive for
> deploying solutions at scale. It clarifies that this interface is a > debug-tool that enterprise environment need not worry about.
>
> The more salient reason for the taint, speaking only for myself as a > Linux kernel community member not for $employer, is to encourage open<= br> > collaboration. Take firmware-update for example that is a standard
> command with known side effects that is inaccessible via the ioctl() > path. It is placed behind an ABI that is easier to maintain and reason=
> about. Everyone has the firmware update tool if they have the 'cat= '
> command. Distros appreciate the fact that they do not need ship yet > another vendor device-update tool, vendors get free tooling and end > users also appreciate one flow for all devices.
>
> As I alluded here [1], I am not against innovation outside of the
> specification, but it needs to be open, and it needs to plausibly beco= me
> if not a de jure standard at least a de facto standard.
>
> [1]: https://lore.kernel.org/all/CAPcyv4gDShAYih5iWabKg_eTHhuHm54vEAei8Zkcm= HnPp3B0cw@mail.gmail.com/

Agree with all this.

>
> > A concern was raised about GAE/FAST/LDST tables for CXL Fabrics > > (a r3.1 feature) but, as I understand it, these are intended for = a
> > host to configure and should not have side effects on other hosts= ?
> > My working assumption is that the kernel driver stack will handle=
> > these (once we catch up with the current feature backlog!) Curren= tly
> > we have no visibility of what the OS driver stack for a fabrics w= ill
> > actually look like - the spec is just the starting point for that=
> > (patches welcome ;)
> >
> > The various CXL upstream developers and maintainers may have
> > differing views of course, but my current understanding is we wan= t
> > to support 1 and 2, but are very resistant to 3!=C2=A0
>
> 1, yes, 2, need to see the patches, and agree on 3.

If we end up with top architecture of the diagrams above, 2 will look prett= y
similar to last version of the switch-cci patches.=C2=A0 So raw commands on= ly + taint.
Factoring out MMPT is another layer that doesn't make that much differe= nce in
practice to this discussion. Good to have, but the reuse here would be one = layer
above that.

Or we just say go for second proposed architecture and 0 impact on the
CXL specific code, just reuse of the MMPT layer.=C2=A0 I'd imagine peop= le will get
grumpy on code duplication (and we'll spend years rejecting patch sets = that
try to share the cdoe) but there should be no maintenance burden as
a result.

>
> > General Notes
> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> >
> > One side aspect of why we really don't like unfiltered usersp= ace access to any
> > of these devices is that people start building non standard hacks= in and we
> > lose the ecosystem advantages. Forcing a considered discussion + = patches
> > to let a particular command be supported, drives standardization.= =C2=A0
>
> Like I said above, I think this is not a side aspect. It is fundamenta= l
> to the viability Linux as a project. This project only works because > organizations with competing goals realize they need some common
> infrastructure and that there is little to be gained by competing on t= he
> commons.
>
> > https://lore.kernel.org/linux-cxl/CAPcyv4gDShAYih5iWabKg_eTHhuH= m54vEAei8ZkcmHnPp3B0cw@mail.gmail.com/
> > provides some history on vendor specific extensions and why in ge= neral we
> > won't support them upstream.=C2=A0
>
> Oh, you linked my writeup... I will leave the commentary I added here = in case
> restating it helps.
>
> > To address another question raised in an earlier discussion:
> > Putting these Fabric Management interfaces behind guard rails of = some type
> > (e.g. CONFIG_IM_A_BMC_AND_CAN_MAKE_A_MESS) does not encourage the= risk
> > of non standard interfaces, because we will be even less likely t= o accept
> > those upstream!
> >
> > If anyone needs more details on any aspect of this please ask. > > There are a lot of things involved and I've only tried to giv= e a fairly
> > minimal illustration to drive the discussion. I may well have mis= sed
> > something crucial.=C2=A0
>
> You captured it well, and this is open source so I may have missed
> something crucial as well.
>

Thanks for detailed reply!

Jonathan



This ele= ctronic communication and the information and any files transmitted with it= , or attached to it, are confidential and are intended solely for the use o= f the individual or entity to whom it is addressed and may contain informat= ion that is confidential, legally privileged, protected by privacy laws, or= otherwise restricted from disclosure to anyone else. If you are not the in= tended recipient or the person responsible for delivering the e-mail to the= intended recipient, you are hereby notified that any use, copying, distrib= uting, dissemination, forwarding, printing, or copying of this e-mail is st= rictly prohibited. If you received this e-mail in error, please return the = e-mail to the sender, delete it from your computer, and destroy any printed= copy of it. --000000000000243380061628359f-- --0000000000002fd1250616283586 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIIQggYJKoZIhvcNAQcCoIIQczCCEG8CAQExDzANBglghkgBZQMEAgEFADALBgkqhkiG9w0BBwGg gg3ZMIIFDTCCA/WgAwIBAgIQeEqpED+lv77edQixNJMdADANBgkqhkiG9w0BAQsFADBMMSAwHgYD VQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMzETMBEGA1UEChMKR2xvYmFsU2lnbjETMBEGA1UE AxMKR2xvYmFsU2lnbjAeFw0yMDA5MTYwMDAwMDBaFw0yODA5MTYwMDAwMDBaMFsxCzAJBgNVBAYT AkJFMRkwFwYDVQQKExBHbG9iYWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBS MyBQZXJzb25hbFNpZ24gMiBDQSAyMDIwMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA vbCmXCcsbZ/a0fRIQMBxp4gJnnyeneFYpEtNydrZZ+GeKSMdHiDgXD1UnRSIudKo+moQ6YlCOu4t rVWO/EiXfYnK7zeop26ry1RpKtogB7/O115zultAz64ydQYLe+a1e/czkALg3sgTcOOcFZTXk38e aqsXsipoX1vsNurqPtnC27TWsA7pk4uKXscFjkeUE8JZu9BDKaswZygxBOPBQBwrA5+20Wxlk6k1 e6EKaaNaNZUy30q3ArEf30ZDpXyfCtiXnupjSK8WU2cK4qsEtj09JS4+mhi0CTCrCnXAzum3tgcH cHRg0prcSzzEUDQWoFxyuqwiwhHu3sPQNmFOMwIDAQABo4IB2jCCAdYwDgYDVR0PAQH/BAQDAgGG MGAGA1UdJQRZMFcGCCsGAQUFBwMCBggrBgEFBQcDBAYKKwYBBAGCNxQCAgYKKwYBBAGCNwoDBAYJ KwYBBAGCNxUGBgorBgEEAYI3CgMMBggrBgEFBQcDBwYIKwYBBQUHAxEwEgYDVR0TAQH/BAgwBgEB /wIBADAdBgNVHQ4EFgQUljPR5lgXWzR1ioFWZNW+SN6hj88wHwYDVR0jBBgwFoAUj/BLf6guRSSu TVD6Y5qL3uLdG7wwegYIKwYBBQUHAQEEbjBsMC0GCCsGAQUFBzABhiFodHRwOi8vb2NzcC5nbG9i YWxzaWduLmNvbS9yb290cjMwOwYIKwYBBQUHMAKGL2h0dHA6Ly9zZWN1cmUuZ2xvYmFsc2lnbi5j b20vY2FjZXJ0L3Jvb3QtcjMuY3J0MDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwuZ2xvYmFs c2lnbi5jb20vcm9vdC1yMy5jcmwwWgYDVR0gBFMwUTALBgkrBgEEAaAyASgwQgYKKwYBBAGgMgEo CjA0MDIGCCsGAQUFBwIBFiZodHRwczovL3d3dy5nbG9iYWxzaWduLmNvbS9yZXBvc2l0b3J5LzAN BgkqhkiG9w0BAQsFAAOCAQEAdAXk/XCnDeAOd9nNEUvWPxblOQ/5o/q6OIeTYvoEvUUi2qHUOtbf jBGdTptFsXXe4RgjVF9b6DuizgYfy+cILmvi5hfk3Iq8MAZsgtW+A/otQsJvK2wRatLE61RbzkX8 9/OXEZ1zT7t/q2RiJqzpvV8NChxIj+P7WTtepPm9AIj0Keue+gS2qvzAZAY34ZZeRHgA7g5O4TPJ /oTd+4rgiU++wLDlcZYd/slFkaT3xg4qWDepEMjT4T1qFOQIL+ijUArYS4owpPg9NISTKa1qqKWJ jFoyms0d0GwOniIIbBvhI2MJ7BSY9MYtWVT5jJO3tsVHwj4cp92CSFuGwunFMzCCA18wggJHoAMC AQICCwQAAAAAASFYUwiiMA0GCSqGSIb3DQEBCwUAMEwxIDAeBgNVBAsTF0dsb2JhbFNpZ24gUm9v dCBDQSAtIFIzMRMwEQYDVQQKEwpHbG9iYWxTaWduMRMwEQYDVQQDEwpHbG9iYWxTaWduMB4XDTA5 MDMxODEwMDAwMFoXDTI5MDMxODEwMDAwMFowTDEgMB4GA1UECxMXR2xvYmFsU2lnbiBSb290IENB IC0gUjMxEzARBgNVBAoTCkdsb2JhbFNpZ24xEzARBgNVBAMTCkdsb2JhbFNpZ24wggEiMA0GCSqG SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDMJXaQeQZ4Ihb1wIO2hMoonv0FdhHFrYhy/EYCQ8eyip0E XyTLLkvhYIJG4VKrDIFHcGzdZNHr9SyjD4I9DCuul9e2FIYQebs7E4B3jAjhSdJqYi8fXvqWaN+J J5U4nwbXPsnLJlkNc96wyOkmDoMVxu9bi9IEYMpJpij2aTv2y8gokeWdimFXN6x0FNx04Druci8u nPvQu7/1PQDhBjPogiuuU6Y6FnOM3UEOIDrAtKeh6bJPkC4yYOlXy7kEkmho5TgmYHWyn3f/kRTv riBJ/K1AFUjRAjFhGV64l++td7dkmnq/X8ET75ti+w1s4FRpFqkD2m7pg5NxdsZphYIXAgMBAAGj QjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBSP8Et/qC5FJK5N UPpjmove4t0bvDANBgkqhkiG9w0BAQsFAAOCAQEAS0DbwFCq/sgM7/eWVEVJu5YACUGssxOGhigH M8pr5nS5ugAtrqQK0/Xx8Q+Kv3NnSoPHRHt44K9ubG8DKY4zOUXDjuS5V2yq/BKW7FPGLeQkbLmU Y/vcU2hnVj6DuM81IcPJaP7O2sJTqsyQiunwXUaMld16WCgaLx3ezQA3QY/tRG3XUyiXfvNnBB4V 14qWtNPeTCekTBtzc3b0F5nCH3oO4y0IrQocLP88q1UOD5F+NuvDV0m+4S4tfGCLw0FREyOdzvcy a5QBqJnnLDMfOjsl0oZAzjsshnjJYS8Uuu7bVW/fhO4FCU29KNhyztNiUGUe65KXgzHZs7XKR1g/ XzCCBWEwggRJoAMCAQICDFYaRLSTNnawblU3yzANBgkqhkiG9w0BAQsFADBbMQswCQYDVQQGEwJC RTEZMBcGA1UEChMQR2xvYmFsU2lnbiBudi1zYTExMC8GA1UEAxMoR2xvYmFsU2lnbiBHQ0MgUjMg UGVyc29uYWxTaWduIDIgQ0EgMjAyMDAeFw0yMjA5MTAwOTI0MjNaFw0yNTA5MTAwOTI0MjNaMIGc MQswCQYDVQQGEwJJTjESMBAGA1UECBMJS2FybmF0YWthMRIwEAYDVQQHEwlCYW5nYWxvcmUxFjAU BgNVBAoTDUJyb2FkY29tIEluYy4xHDAaBgNVBAMTE1NyZWVuaXZhcyBCYWdhbGtvdGUxLzAtBgkq hkiG9w0BCQEWIHNyZWVuaXZhcy5iYWdhbGtvdGVAYnJvYWRjb20uY29tMIIBIjANBgkqhkiG9w0B AQEFAAOCAQ8AMIIBCgKCAQEAuuAqYt2XMpXqqYrSYVv64PlRiKuXJ1Hesi3rmZm4+g8EIZRtyETO RWiFc6q59aDOqgjp94efpIVthPgLGM5Uv2iBj+XGsZzaiJn4ZasS5UNh0N6Rpj6dLHXBwRGaQsld e5WvrXWFBwSZWVNj241paUqCIV+ybg8DS6/tCwEetzOgmjs4LgYlpj957G5EryaCzH5ncDrHz2Kj q98WQ9dndZnpUpaPV2hP60IVspaStey0I6WjPBy7qNhbZV4J4ZN71tnZ7CrNkXy5KhwKEs+MiegU OQvZ9yJQdMq9Ry3cO3DNfLXNGHuM2IKh6YPPScUnfGbmK/wkpXthxhJx+kn1IQIDAQABo4IB4TCC Ad0wDgYDVR0PAQH/BAQDAgWgMIGjBggrBgEFBQcBAQSBljCBkzBOBggrBgEFBQcwAoZCaHR0cDov L3NlY3VyZS5nbG9iYWxzaWduLmNvbS9jYWNlcnQvZ3NnY2NyM3BlcnNvbmFsc2lnbjJjYTIwMjAu Y3J0MEEGCCsGAQUFBzABhjVodHRwOi8vb2NzcC5nbG9iYWxzaWduLmNvbS9nc2djY3IzcGVyc29u YWxzaWduMmNhMjAyMDBNBgNVHSAERjBEMEIGCisGAQQBoDIBKAowNDAyBggrBgEFBQcCARYmaHR0 cHM6Ly93d3cuZ2xvYmFsc2lnbi5jb20vcmVwb3NpdG9yeS8wCQYDVR0TBAIwADBJBgNVHR8EQjBA MD6gPKA6hjhodHRwOi8vY3JsLmdsb2JhbHNpZ24uY29tL2dzZ2NjcjNwZXJzb25hbHNpZ24yY2Ey MDIwLmNybDArBgNVHREEJDAigSBzcmVlbml2YXMuYmFnYWxrb3RlQGJyb2FkY29tLmNvbTATBgNV HSUEDDAKBggrBgEFBQcDBDAfBgNVHSMEGDAWgBSWM9HmWBdbNHWKgVZk1b5I3qGPzzAdBgNVHQ4E FgQU08YdPZld7W8H+uK8S2EPeYqBn20wDQYJKoZIhvcNAQELBQADggEBAGUBviL4v9e/8GOQBEZO mm5a/OZCkJ6COPgkQ4TEHxWWx7iYG+TVkJ2qxNS1fXQUPnAC/TBfb37wLVynlBBRTb4PYzKIfaHW bH9AscUsus4kEUyzYSLHYqQilComzpGFxor1x2fbLYpHMCU950wueyUHPHA3jwpWrluCE7283dWT rz/VNFx2cjW+y8+T+lVxO44XmrQoa8lnz3fb+QCVvRyH8sUqAYjfkscFkknlkF0NobKXFSKwK1zZ 4ZQOk9tT7FYGWvy0nhOEUt/uKZNncumowkhquxqJve0HiOF6Z6R0eKybmGDGaWDYE11aHV62bKTn /SH9BOZuA2GG8ZiNR08xggJtMIICaQIBATBrMFsxCzAJBgNVBAYTAkJFMRkwFwYDVQQKExBHbG9i YWxTaWduIG52LXNhMTEwLwYDVQQDEyhHbG9iYWxTaWduIEdDQyBSMyBQZXJzb25hbFNpZ24gMiBD QSAyMDIwAgxWGkS0kzZ2sG5VN8swDQYJYIZIAWUDBAIBBQCggdQwLwYJKoZIhvcNAQkEMSIEIK9Q 6ZWXQsHaM2nHvJ8LEaYFVlEzPUC6niCVh8RXvMBlMBgGCSqGSIb3DQEJAzELBgkqhkiG9w0BBwEw HAYJKoZIhvcNAQkFMQ8XDTI0MDQxNTIwMTAwN1owaQYJKoZIhvcNAQkPMVwwWjALBglghkgBZQME ASowCwYJYIZIAWUDBAEWMAsGCWCGSAFlAwQBAjAKBggqhkiG9w0DBzALBgkqhkiG9w0BAQowCwYJ KoZIhvcNAQEHMAsGCWCGSAFlAwQCATANBgkqhkiG9w0BAQEFAASCAQAp2+MuO1iKZG6aLtbxwC2p 5DP1sl0qDgNdAD71LS5MxZNS4JiV0dnq9UyYzEKZAAelVrxhq0tpzw+47wHZ07wB+//jWNwFEYfr 2equx7IEdHdoEPMVMCQQd0q0vhVNrFTgECpy2QPZgrTNGTFLOieMJUSilFBDW8XlMTpnZyZCcome zoGK70QwJK76h6kZkW2Brt47wGwfrYAw1cxukq6G85WQFyBWZ1NUzUgUzDXVqV1XYUiUZZuHLzdb FVzKThoW48AtRYidYy8VciqnJ3CXzDDp+dxUeRJxYiAim1lNBRTmsKczGw9+1YBEAwOig8lq7owo indv7MtM8B89TazD --0000000000002fd1250616283586--