[OSSA-2017-002] Failed notification payload is dumped in logs with auth secrets (CVE-2017-7214)

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

If it can be confirmed for certain that the commit introducing this behavior is master-branch-only impacting then we can drop the embargo and forgo the advisory (Class Y report).

Hmm, so the problem is that the notifier is hooked directly into the logger, and when it sends the notification it tries to include all of the args to the logger. In this case that includes the Nova context, which contains sensitive data.

The circular reference error aside, it seems bad that we're sending sensitive data in notification payloads. That isn't considered a secure channel, is it?

Maybe we could filter the context out in the notification log handler. I don't think that information is intended to be exposed, except where it is referenced in the log message anyway.

I'm pretty sure notifications aren't considered a secure channel. If you're sending over rpc you should have ssl configured. But it's not required, and I remember long ago when the IBM product I was working on was using QEMU we were getting internal security audit flags because QEMU was logging credentials that were sent over the wire.

We have a fix here:

https://review.openstack.org/#/c/446948/

Looks like the original change to strip the context out of the notification payload happened in Grizzly:

https://github.com/openstack/nova/commit/d1f121265badfffd097ee983d14989b885375a0e

And regressed in Mitaka:

https://github.com/openstack/nova/commit/2910d75b28afd909af3b4ac392729ac3d5e64b65

The actual Tempest aggregate negative tests are very old:

https://github.com/openstack/tempest/commit/bf2d51751d778eb7a1b9ddfe1901f15c4cf56192

So why things started getting logged recently I'm still not sure, but my guess would be something changed in oslo.messaging to be logging on failures.

https://review.openstack.org/#/c/309679/ in oslo.messaging might have started logging the failure, but that's been around since Newton so it's very odd that we'd just start noticing this now.

All of the backports are up too:

https://review.openstack.org/#/q/I1d217620e52d45595a3e0e49ed57b4ab33cd1688,n,z

Maybe there was a change to the context that caused the circular reference error? I see a comment added in https://github.com/openstack/oslo.context/commit/f25543fcc792ebf155728a91fde06e8dc4e96cea that refers to avoiding circular references, but maybe it's not working in this case.

That didn't merge until January, so it's possible nobody noticed until now.

It looks like the backports are starting to get approved and merge now as well. Once they're all merged, I propose switching this bug to Public Security and following our lighter-weight our public report disclosure process from that point instead of continuing to maintain an embargo (the risk posed by this bug seems pretty minimal).

Proposed impact description:

Title: Nova logs sensitive context from notification exceptions
Reporter: Matt Riedemann (Huawei)
Products: Nova
Affects: >=13.0.0 <=13.1.3, >=14.0.0 <=14.0.4, >=15.0.0 <=15.0.1

Description:
Matt Riedemann with Huawei reported a vulnerability in Nova. Legacy notification exception contexts appearing in ERROR level logs may include sensitive information such as account passwords and authorization tokens. All Nova setups are affected.

impact statement looks good to me and +1 making it public for the reasons fungi described.

Impact statement looks good to me too.

The nova fixes are merged on all branches for this now. Can we make it public security and then I can call it out in the release notes for the nova changes on stable and do a release?

Thanks Matt, I've switched this to Public Security and am moving forward with the CVE assignment request using the impact statement from comment #10.

Removed unneeded security tag, as the bug type is already security (we use the former for hardening opportunities and the latter for vulnerabilities).

Related fix proposed to branch: master
Review: https://review.openstack.org/448295

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/448297

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/448298

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/448299

Reviewed: https://review.openstack.org/448295
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3f985f1eda6f29180878a3d21c20c5057179486a
Submitter: Jenkins
Branch: master

commit 3f985f1eda6f29180878a3d21c20c5057179486a
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569

Reviewed: https://review.openstack.org/448297
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=acb19160d4d348e29a21ad57c61c7369352c4d1c
Submitter: Jenkins
Branch: stable/ocata

commit acb19160d4d348e29a21ad57c61c7369352c4d1c
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)

Reviewed: https://review.openstack.org/448298
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a
Submitter: Jenkins
Branch: stable/newton

commit c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)
    (cherry picked from commit acb19160d4d348e29a21ad57c61c7369352c4d1c)

Reviewed: https://review.openstack.org/448299
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e193201fa1de5b08b29adefd8c149935c5529598
Submitter: Jenkins
Branch: stable/mitaka

commit e193201fa1de5b08b29adefd8c149935c5529598
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)
    (cherry picked from commit acb19160d4d348e29a21ad57c61c7369352c4d1c)
    (cherry picked from commit c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a)

Reviewed: https://review.openstack.org/448610
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=c54ed705df6588194d134abe1762aae5e0e5a39b
Submitter: Jenkins
Branch: master

commit c54ed705df6588194d134abe1762aae5e0e5a39b
Author: Jeremy Stanley <email address hidden>
Date: Wed Mar 22 14:23:58 2017 +0000

    OSSA-2017-002 (CVE-2017-7214)

    Nova logs sensitive context from notification exceptions

    Change-Id: Iec1deae6bbe7fc73045c2abf9b3d44bafa86acc0
    Closes-Bug: #1673569

Hello Matt, or anyone else affected,

Accepted nova into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:14.0.5-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Matt, or anyone else affected,

Accepted nova into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

SRU regression testing has completed successfully for yakkety-newton and xenial-newton. Tempest results are below:

yakkety-newton-proposed
-----------------------

======
Totals
======
Ran: 103 tests in 972.3608 sec.
 - Passed: 97
 - Skipped: 6
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 554.7547 sec.

==============
Worker Balance
==============
 - Worker 0 (103 tests) => 0:15:59.401528
___________________________________ summary ____________________________________
  smoke: commands succeeded
  congratulations :)

xenial-newton-proposed
----------------------

======
Totals
======
Ran: 103 tests in 1185.2261 sec.
 - Passed: 97
 - Skipped: 6
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 650.9384 sec.

==============
Worker Balance
==============
 - Worker 0 (103 tests) => 0:19:27.568502
___________________________________ summary ____________________________________
  smoke: commands succeeded
  congratulations :)

This bug was fixed in the package nova - 2:14.0.5-0ubuntu1

---------------
nova (2:14.0.5-0ubuntu1) yakkety; urgency=medium

  [Saverio Proto]
  * New upstream point release for OpenStack Newton (LP: #1688557).

  [Corey Bryant]
  * SECURITY UPDATE: Failed notification payload is dumped in logs
    with auth secrets (LP: #1673569).
    - This is included from upstream in the 14.0.5 stable point release.
    - CVE-2017-7214

 -- Corey Bryant <email address hidden> Mon, 15 May 2017 08:37:32 -0400

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:14.0.5-0ubuntu1~cloud0
---------------

 nova (2:14.0.5-0ubuntu1~cloud0) xenial-newton; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (2:14.0.5-0ubuntu1) yakkety; urgency=medium
 .
   [Saverio Proto]
   * New upstream point release for OpenStack Newton (LP: #1688557).
 .
   [Corey Bryant]
   * SECURITY UPDATE: Failed notification payload is dumped in logs
     with auth secrets (LP: #1673569).
     - This is included from upstream in the 14.0.5 stable point release.
     - CVE-2017-7214