KVM: Enable Secure Execution Crypto Passthrough - qemu part

Bug #2003673 reported by bugproxy
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
In Progress
High
Skipper Bug Screeners
qemu (Ubuntu)
In Progress
High
Sergio Durigan Junior
Mantic
In Progress
High
Sergio Durigan Junior

Bug Description

SRU Justification:

[ Impact ]

 * As of today IBM Z crypto hardware (AP, Adjunct Processor, an IBM Z cryptographic facility, that provides cryptographic virtual functions to Linux on Z system) cannot be passed-through to Secure Execution guests (pv-guests) by KVM and QEMU. (All this in context of confidential computing aka secure execution.)

 * But it's highly desired to make use of APs from within Secure Execution guests, since they make significant use of cryptography in general, hence this should be ideally off-loadable and supported by hardware cryptographic (APs).

 * Enabling KVM and QEMU for AP passthrough to Secure Execution guests includes setup, configuration and tear-down of AP related resources and affects Kernel, qemu (and s390-tools).

 * QEMU has to turn on the corresponding bits in the KVM CPU-model if the CPU firmware supports it.

 * However, it only makes sense to turn on AP-PT if the QEMU user enabled (general) AP for that guest.

 * The kernel adjustments (which is the bigger set of changes with 20+ commits) made it in time for mantic into the Ubuntu kernel 6.5 (backports from 6.6), but not the qemu changes.

 * This SRU seeks an exception, to add the needed qemu changes post-GA to bring this overall functionality to work, since it's desired to exploit this as part of a product.

[ Fixes ]

 * 297ec01f0b s390x/ap: fix missing subsystem reset registration

 * ef1535901a s390x: do a subsystem reset before the unprotect on reboot

 * da3c22c74a linux-headers: Update to Linux v6.6-rc1

 * 354383c122 target/s390x/kvm: Refactor AP functionalities

 * 5ac951519c target/s390x: AP-passthrough for PV guests

[ Test Plan ]

 * Have an Ubuntu 23.10 LPAR with at least one (better several) APs assigned. Check with:
   $ lszcrypt -V
   CARD.DOM TYPE MODE STATUS REQUESTS PENDING HWTYPE QDEPTH FUNCTIONS DRIVER
   --------------------------------------------------------------------------------------------
   00 CEX7C CCA-Coproc online 1 0 11 08 S--D--N--- cex7card
   00.000a CEX/C CCA-Coproc online 1 0 11 08 S--D--N--- cex7queue

 * Setup a secure execution environment according to
   'Introducing IBM Secure Execution for Linux' (SC34-7721-03)
   See PDF page #15 ff, that use Ubuntu as an example.

 * Pass selected AP resources from the KVM host through to the Secure Execution (SE) guest.

 * Enable the SE guest to make use of hardware cryptographic facilities by installing the needed packages:
   $ sudo apt-get install libica-utils libica? openssl-ibmca
   and configure openssl:
   $ sudo cp /usr/share/doc/openssl-ibmca/examples/openssl.cnf.sample /etc/ssl/openssl.cnf
   This allows to make use of the ibmca openssl engine:
   $ openssl engine -c
   (dynamic) Dynamic engine loading support
   (ibmca) Ibmca hardware engine support
    [RSA, DSA, DH, RAND, DES-ECB, DES-CBC, DES-OFB, DES-CFB, DES-EDE3, DES-EDE3-CBC, DES-EDE3-OFB,
    DES-EDE3-CFB, AES-128-ECB, AES-192-ECB, AES-256-ECB, AES-128-CBC, AES-192-CBC, AES-256-CBC,
    AES-128-OFB, AES-192-OFB, AES-256-OFB, AES-128-CFB, AES-192-CFB, AES-256-CFB, SHA1, SHA256, SHA512]

 * Reset the icastats crypto counters with (just in case they are not zero):
   $ icastats -r

 * Consume cryptographic resources in the secure guest that are known to be executed
   by the underlying and passed through hardware cryptography.
   For example:
   $ openssl speed -evp aes-256-cbc 2>/dev/null | tail -n 3
   will provide this output:
   The 'numbers' are in 1000s of bytes per second processed.
   type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
   AES-256-CBC 160454.13k 510929.25k 1272902.21k 1590194.94k 1708914.64k 1702523.78k
   and will especially lead to icastats counter entries (indicating hardware use) for DRBG-SHA-512:
   $ icastats | grep DRBG-SHA-512
   DRBG-SHA-512 | 175 | 0
   [
   Alternatively use ssh to consume hardware crypto, like with (do an 'icastats -r' before):
   $ ssh -c <email address hidden> ubuntu@localhost
   that will lead to these counter entries:
   $ icastats | grep 'DRBG-SHA-512\|RSA-ME'
   DRBG-SHA-512 | 171 | 0
         RSA-ME | 1 | 0
   ]

 * The verification needs to be done by IBM.

[ Where problems could occur ]

 * Commits 297ec01f0b and ef1535901a are very small fixes (each one line),
   that are pretty traceable.

 * Commit da3c22c74a introduces the needed kernel 6.6 header changes.
   These fit to the 23.10 kernel (since they were already cherry picked: LP#2003674)
   They touch common code files and can potentially affect other architectures.
   However, the majority of modification are additional definitions.

 * Commits 354383c122 touches the s390x target only,
   refactors AP, and changes check, test and set functions.
   Hence if done errornously this can lead to a wrong states.
   Or if the refactoring wasn't done complete functions might be missed,
   but this of course is checked by a test build.
   Overall this does not look too invasive.

 * Commit 5ac951519c, again touching s390x code only,
   is the actual AP-passthrough for PV guests.
   It adds this feature similar to other the existing ones, and expands the CPU model.
   No code is removed, but only added to existing functions and a new set of
   functions got introduced: uv_feat_supported, query_uv_feat_guest, configure_uv_feat_guest
   In case of issues with these, query or config of features can be harmed
   which might lead to wrong states or even break the code.

[ Other Info ]

 * It's for our partner IBM an important new feature (as confidential computing is in general),
   which unfortunately landed late in qemu.

 * It's with 5 patches (with two of them one line fixes) not very invasive.

 * The modification can be and will be thoroughly tested.
__________

Feature Description:

Enable KVM and QEMU for AP passthrough to Secure Execution guests. This includes setup, configuration and teardown of AP related resources.

Related branches

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-201348 severity-high targetmilestone-inin2304
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Frank Heimes (fheimes) wrote :

Is the Secure Execution Crypto Passthrough feature included in upstream qemu 7.2 ?

affects: linux (Ubuntu) → qemu (Ubuntu)
tags: added: qemu-23.04
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
importance: Undecided → High
Changed in qemu (Ubuntu):
importance: Undecided → High
status: New → Incomplete
Changed in ubuntu-z-systems:
status: New → Incomplete
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Any updates what happened to this one?

tags: added: qemu-23.10
removed: qemu-23.04
Revision history for this message
Frank Heimes (fheimes) wrote :

This ticket is part of a larger feature known as HSM for secure execution.
The overall HSM4SE feature slipped to 23.10, hence retagging to 23.10 is correct.
Sorry that it was forgotten to update the tag after lunar GA.

summary: - [23.04 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part
+ [23.10 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part
bugproxy (bugproxy)
tags: added: targetmilestone-inin2310
removed: targetmilestone-inin2304
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-08-16 11:52 EDT-------
This feature is not yet available. With missing Mantic FF, we need to postpone it to the next Ubuntu release.

==> Changing target milestone to: "24.04"

tags: added: targetmilestone-inin2404
removed: targetmilestone-inin2310
Frank Heimes (fheimes)
summary: - [23.10 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part
+ [24.04 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part
tags: added: qemu-24.04
removed: qemu-23.10
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-09-03 18:32 EDT-------
In the meantime, the QEMU patches are in a pretty stable state (see https://lists.gnu.org/archive/html/qemu-devel/2023-08/msg03015.html)
and are basically only waiting for the related kernel patches of this feature to be pulled before they can go into the QEMU repository.

The related bugzilla / LP entry covering the kernel part is the following:
Bug 201347 - LP2003674 : [23.10 FEAT] [VS2207] KVM: Enable Secure Execution Crypto Passthrough - kernel part )

Revision history for this message
Frank Heimes (fheimes) wrote : Re: [24.04 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part

Since this became increasingly important (raised by IBM)
and we were asked to have that ideally in 23.10
and the needed kernel parts will also highly likely make it into mantic (submission to the kernel teams mailing list was done based on LP#2003674),
I'm transforming this now into a FFe,
so that in case the commits get upstream accepted soon-ish / in time,
we "can think of and try" to get them into mantic.
Final decision is up to the release team (as usual with FFe-s).

Since this is a bigger KVM/Secure-Execution feature (crypto passthrough),
I believe there is value to have this already included in 23.10 now,
so that it can settle and stabilize for the next LTS.

There is also a team that plans to use this as part of a product.

summary: - [24.04 FEAT] KVM: Enable Secure Execution Crypto Passthrough - qemu part
+ [FFe] KVM: Enable Secure Execution Crypto Passthrough - qemu part
Changed in ubuntu-z-systems:
status: Incomplete → New
Changed in qemu (Ubuntu):
status: Incomplete → New
Revision history for this message
Frank Heimes (fheimes) wrote : Re: [FFe] KVM: Enable Secure Execution Crypto Passthrough - qemu part

Kernel patches in LP#2003674 just got applied, means accepted for mantic (in master-next now).

Revision history for this message
Frank Heimes (fheimes) wrote :

FFe is no option anymore, we need to go with an SRU post-GA.

summary: - [FFe] KVM: Enable Secure Execution Crypto Passthrough - qemu part
+ KVM: Enable Secure Execution Crypto Passthrough - qemu part
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2023-10-18 09:07 EDT-------
People have been prodding me to get a status for the post-ga release.

Could you please give us a possible timeline for a post-ga release?

Revision history for this message
Frank Heimes (fheimes) wrote :

Hello Frank, as a pre-req. for SRU-ing the commits need to be upstream accepted.
Based on comment #5 the following commits are requested for inclusion:
[PATCH v3 1/5] s390x/ap: fix missing subsystem reset registration
[PATCH v3 2/5] s390x: switch pv and subsystem reset ordering on reboot
[PATCH v3 3/5] NOTFORMERGE update linux-headers/asm-s390/kvm.h
[PATCH v3 4/5] target/s390x/kvm: Refactor AP functionalities
[PATCH v3 5/5] target/s390x: AP-passthrough for PV guests,
whereas 3/5 can be ignored.

Last time (and it was prior to 23.10 feature freeze) I couldn't find the commits in master, so they weren't upstream accepted at that time.

Today I can still not find "s390x: switch pv and subsystem reset ordering on reboot".
Is this one required, was it skipped and is no longer needed or will it be accepted later?
(I wasn't able to find more conversation about this on qemu-devel other than:
https://lists.gnu.org/archive/html/qemu-devel/2023-08/msg03015.html)

Once everything needed is upstream accepted, we can start to work on the SRU.
(And will probably need your input for the SRU justification, esp. for the test plan section.)

SRU's are processed on a queue base, hence it is not easily possible to give you a target date (it's different that with kernel SRUs).

What we could do (even before upstream acceptance) is to generate a qemu test build in PPA that incl. these commits - in case that helps.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-10-18 10:15 EDT-------
The patches have been pulled and are on main/master:
https://<email address hidden>/

The missing patch might have been renamed a tiny bit and hence it might have slipped through when checking for it.

My first name is Janosch :)

Revision history for this message
Frank Heimes (fheimes) wrote :

Hi Janosch, oh I'm very sorry - that was probably because my brain is so trained with 'Frank' (for an obvious reason ;-).

So the commits needed as patches are:
297ec01f0b s390x/ap: fix missing subsystem reset registration
ef1535901a s390x: do a subsystem reset before the unprotect on reboot
da3c22c74a linux-headers: Update to Linux v6.6-rc1
354383c122 target/s390x/kvm: Refactor AP functionalities
5ac951519c target/s390x: AP-passthrough for PV guests
4ddf7728fb kconfig: Add NVME to s390x machines
82fdcd3e14 tests/qtest/pflash: Clean up local variable shadowing

(I wasn't aware about this updated patch set, I was still looking for the v3 set.)

With that we can start with the SRU (hoping that the patches can be cleanly cherry-picked into mantic's qemu 8.0.4).

Revision history for this message
Frank Heimes (fheimes) wrote :

Having a closer look at the above commits I think 4ddf7728fb and 82fdcd3e14 from the overall PR are not needed to enable secure Eeecution crypto passthrough, and got just incl. in the PR.
Since we are asked to keep modifications introduced by SRUs to a bare minimum, it's better to skip these.
Please can you confirm that these are not required for this?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2023-10-19 02:35 EDT-------
These two patches are NOT needed for AP passthrough support:

4ddf7728fb kconfig: Add NVME to s390x machines
82fdcd3e14 tests/qtest/pflash: Clean up local variable shadowing

Frank Heimes (fheimes)
description: updated
description: updated
Frank Heimes (fheimes)
information type: Private → Public
tags: added: server-todo
Changed in qemu (Ubuntu Mantic):
assignee: Skipper Bug Screeners (skipper-screen-team) → Sergio Durigan Junior (sergiodj)
Frank Heimes (fheimes)
Changed in qemu (Ubuntu Mantic):
status: New → In Progress
Changed in qemu (Ubuntu):
status: New → In Progress
Changed in ubuntu-z-systems:
status: New → In Progress
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Hi,

We had our Engineering Sprint last week, so I didn't have time to work on this. I'm back now, and have made good progress on investigating the testsuite failures I've been seeing. So far, all failures have been false positives which makes me confident that the patchset is OK. I should be able to finish the investigation in the next couple of days, and then will proceed with SRU.

Thank you for the patience.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.