Message-ID: <e37fe712-4e0e-229a-a07e-52a0d486819c@nvidia.com>
Date:   Mon, 23 Jan 2023 14:57:18 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101
 Thunderbird/102.6.1
Subject: Re: [PATCH rdma-next 00/13] Add RDMA inline crypto support
To:     Sagi Grimberg <sagi@grimberg.me>,
        Leon Romanovsky <leon@kernel.org>,
        Jason Gunthorpe <jgg@nvidia.com>
Cc:     Bryan Tan <bryantan@vmware.com>, Christoph Hellwig <hch@lst.de>,
        Eric Dumazet <edumazet@google.com>,
        Jakub Kicinski <kuba@kernel.org>, Jens Axboe <axboe@fb.com>,
        Keith Busch <kbusch@kernel.org>, linux-kernel@vger.kernel.org,
        linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
        linux-trace-kernel@vger.kernel.org,
        Masami Hiramatsu <mhiramat@kernel.org>,
        Max Gurtovoy <mgurtovoy@nvidia.com>, netdev@vger.kernel.org,
        Paolo Abeni <pabeni@redhat.com>,
        Saeed Mahameed <saeedm@nvidia.com>,
        Selvin Xavier <selvin.xavier@broadcom.com>,
        Steven Rostedt <rostedt@goodmis.org>,
        Vishnu Dasa <vdasa@vmware.com>,
        Yishai Hadas <yishaih@nvidia.com>
References: <cover.1673873422.git.leon@kernel.org>
 <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me>
Content-Language: en-US
From:   Israel Rukshin <israelr@nvidia.com>
In-Reply-To: <6f9da88c-f01e-156b-eb19-0b275c46c6b5@grimberg.me>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0
X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1
X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R0VzSjlXN0VsbUtwclF0c1AyZGJnQTJrUFlGN1FoNGZzMGd5SlBFZUpMQTFk?=
 =?utf-8?B?OURvZ1pjTTh0VGJtNnR2bEIrUHlkNGppQjd2VXVTOEMxa0pWRDh5Tm1xNXBa?=
 =?utf-8?B?LzdiaFBScU1ycCtqcW5XZk5TdVZkT2wxT2M0U1cwN0hvQllySC9jbVFVK3JV?=
 =?utf-8?B?T1l0d1RlU0duMWV2ajFjZVBRSzFuRUxtK3p4bExaWFAwOG9DbEpwZllTY3Za?=
 =?utf-8?B?cWxNSDc5THBNdXR4b25Db05KeHB4b0RyWGFXd1JEdzFOVXVUS3pVU01KS0FV?=
 =?utf-8?B?eWQvcVlidEV0Z2JDc1pxWUZxdGxtd052TlBybm5qK08zZHJmYW8weFl2aGxN?=
 =?utf-8?B?aG92aVdnRDJudk9rK0s0RkdBUTQzaVZsZVMyRXlXbXlsODFWdHp0K3lhQjdq?=
 =?utf-8?B?SFUzN1d5dlRYN2NxUStrQ2VZQUFjaXkyaHpER0pGbkc2YmJhODluYVk2d1Vh?=
 =?utf-8?B?bnA2NHN3dlRFYXBWbWhHODVOZExHdHFwRFhCVjdnMDJWMVNTaFo1Ky9ha2k1?=
 =?utf-8?B?QlZCVHI0SXdIQ2t5UWpCUFgrbGVCL0IzR3JUSk9Kd1BzQURReG0rK3Buam9o?=
 =?utf-8?B?RzBMUk96bXJrNWJaVFRLeVFXNm1UQllMeWRjQzhlR1ZIazdDYlFpZU1TTHl6?=
 =?utf-8?B?Lys4cDRFKzRSRFJxR3JuRXhOU1VmRVRRMW1QcUdDME5lYnNJRkRKQUQ3RFRO?=
 =?utf-8?B?TERBcnZwZGR6OTdIb2hZRHVHdDgvY2swMjV0U3htc3BDdHhvUUdhUkh4WXlJ?=
 =?utf-8?B?bXNTUmVxVnErQ1V6emlBa0ZLZzNPYWUyR0toa0lhRThXNEdMaGJmSXlQU0Nr?=
 =?utf-8?B?dExWTVFqVFBCcThoV1d2bVhlWnpEQkRNeW02YTZxa0F1RVBNMlYzd2pQWEZB?=
 =?utf-8?B?eWwwMVpCZFJ0V1NwRWpkOEdORG5tZTRKanVZb2NUM0wrMFpuNmpRUXRFQmI5?=
 =?utf-8?B?YnIwamtIQlVuMHFia1BkdzliZmVjRVNEMkxBZ0tIOHNZZVdSVUk3NzhFdStE?=
 =?utf-8?B?VDFkdUtXYmJSQXRCem9sUWpRSTFCWlVVajdQVEVrZmtYMnhMSTdHZ2s4dHlM?=
 =?utf-8?B?M1I0RU45RmEvZTFnT05naUFBd1E0dTVLNmVoaTJNMFo1cEJNQkxXMmlUNTJz?=
 =?utf-8?B?RjFYQ2w1TUE4VWVFd0M2RnBXYzhYbzZJK3RVaStUQktYSERjdmxZVlA0V2Qr?=
 =?utf-8?B?QUNrNXozVnNjYTlMU3k2NFNaNGMxYm5vOXd2dTBoK01LU0J3QWpoMFVGYWtu?=
 =?utf-8?B?cmJSdmd5YkxVRmVCV2diSk56Q3cxakxnVUdkY01jSmFwL1NNQlptUEQ5dVpU?=
 =?utf-8?B?MWxHbCtVTGZCekZCaEo1TGhVYUpSWGtrTHBQdjNCVmVWWXVKKzlXNGg5eC8z?=
 =?utf-8?B?TkxCdUNzcklaUjlVRW9TVVNXdVp4VCtrWXl4N29vWWx4SjlSNEYxdGhrcXJh?=
 =?utf-8?B?TXVKNFNjb0RIdUpBbXZ6TWd4QXpPZk1zWHRKSVZ5eTMrUTJTczRMQkp3L3Nz?=
 =?utf-8?B?WjBjdUx1d2lGUHFOWnErQ3B6VWFSS20yVjhjRGJLQUhDRXprRnkraS9SY21T?=
 =?utf-8?B?bGhIeng3K0lPbTZXQXRjdDNRSTkvVWFhbzdtWHc5dTJaWDA4cStlOTZHMUdS?=
 =?utf-8?B?VG16NzRxblg0ejYxY25nQzQxZm9HSnlSZzlLbkUzTUU3cW5FYlNtRzhrL3ZV?=
 =?utf-8?B?RGhPclk4MUdGZUZ5c21qaDRvTHJkdnYzc3pHSzlia1hFTlFSTFJ6bDJ0U1pX?=
 =?utf-8?B?dVpEY25zR1V6aTVRcjJmQ0g5dDZCbUt1c3pkK2tCN0htR1ZsdGlhTkNuNlR0?=
 =?utf-8?B?OU9wUzNFZFlCYVNoTHNRYnlqOVBkcS9JTVk1NGF4TW5wU0ZVMEdXbTcwKy9z?=
 =?utf-8?B?eWp5WkhRaWtqQ255RisrcU9RazJjUHU1ZFBWZy9QYmNyaUpXY2luc0VLL3hz?=
 =?utf-8?B?NFA1TmlFeWR2ZU9XNklrZVBENUExdnNuS3dkMVdvdnhhcHZnQSs3UmI1aEps?=
 =?utf-8?B?QXlJblNLbjNVZTI4ZWUxRmhYOVpLMlpZNVVHQk44RHFueE93TkxMK2FTWEZv?=
 =?utf-8?B?dVlQb3llOU9PYlZqVWR4Tld4UTNSTWpTT1VzMFhZUHlyTTNoV2x0RHRCalAr?=
 =?utf-8?Q?p1ZrOtVKx6m2O0KpptCXRmnDu?=
X-OriginatorOrg: Nvidia.com
X-MS-Exchange-CrossTenant-Network-Message-Id: b6380cc0-602c-4a8b-1dbe-08dafd41629e
X-MS-Exchange-CrossTenant-AuthSource: MW3PR12MB4409.namprd12.prod.outlook.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Jan 2023 12:57:30.0108
 (UTC)
X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted
X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a
X-MS-Exchange-CrossTenant-MailboxType: HOSTED
X-MS-Exchange-CrossTenant-UserPrincipalName: gXI5E2EUx/sl4Q7tXzvGLBcxsyOJZpPcPYYi5ldrRXHlKWn8a0YKfselsVnifixPznHCQiVANqPBrmGcQFZpvg==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR12MB5438
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

Hi Sagi,

On 1/23/2023 1:27 PM, Sagi Grimberg wrote:
>
>>  From Israel,
>>
>> The purpose of this patchset is to add support for inline
>> encryption/decryption of the data at storage protocols like nvmf over
>> RDMA (at a similar way like integrity is used via unique mkey).
>>
>> This patchset adds support for plaintext keys. The patches were tested
>> on BF-3 HW with fscrypt tool to test this feature, which showed reduce
>> in CPU utilization when comparing at 64k or more IO size. The CPU 
>> utilization
>> was improved by more than 50% comparing to the SW only solution at 
>> this case.
>>
>> How to configure fscrypt to enable plaintext keys:
>>   # mkfs.ext4 -O encrypt /dev/nvme0n1
>>   # mount /dev/nvme0n1 /mnt/crypto -o inlinecrypt
>>   # head -c 64 /dev/urandom > /tmp/master_key
>>   # fscryptctl add_key /mnt/crypto/ < /tmp/master_key
>>   # mkdir /mnt/crypto/test1
>>   # fscryptctl set_policy 152c41b2ea39fa3d90ea06448456e7fb 
>> /mnt/crypto/test1
>>     ** “152c41b2ea39fa3d90ea06448456e7fb” is the output of the
>>        “fscryptctl add_key” command.
>>   # echo foo > /mnt/crypto/test1/foo
>>
>> Notes:
>>   - At plaintext mode only, the user set a master key and the fscrypt
>>     driver derived from it the DEK and the key identifier.
>>   - 152c41b2ea39fa3d90ea06448456e7fb is the derived key identifier
>>   - Only on the first IO, nvme-rdma gets a callback to load the 
>> derived DEK.
>>
>> There is no special configuration to support crypto at nvme modules.
>
> Hey, this looks sane to me in a very first glance.
>
> Few high level questions:
> - what happens with multipathing? when if not all devices are
> capable. SW fallback?
SW fallback happens every time the device doesn't support the specific 
crypto request (which include data-unit-size, mode and dun_bytes).
So with multipathing, one path uses the HW crypto offload and the other 
one uses the SW fallback.
> - Does the crypt stuff stay intact when bio is requeued?
Yes, the crypto ctx is copied when cloning the bio.
>
> I'm assuming you tested this with multipathing? This is not very
> useful if it is incompatible with it.
Yes, sure.
You can see the call to  blk_crypto_reprogram_all_keys(), which is 
called when the controller was reconnected after port toggling.

- Israel