Multiple services exposed over network / firewall refactor

We have 'REJECT' rules in the end of INPUT and FORWARD chains:

[root@nailgun ~]# iptables -L INPUT -n -v | tail -2
    0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
    0 0 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* 999 iptables denied */ limit: avg 5/min burst 5 LOG flags 0 level 7 prefix "iptables denied: "
[root@nailgun ~]# iptables -L FORWARD -n -v | tail -2
13927 45M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 051 forward admin_net conntrack */ ctstate RELATED,ESTABLISHED
    0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

But as far as I know it came to Fuel with CentOS 7 support, so I'm also for setting default 'DROP' policy explicitly (by puppet) for those chains.

>1. Allow all necessary services to be reached only from ADMIN/PXE network segment (ADMIN/PXE network IP addressing as provided by user during Fuel menu installation phase). All other network segments should not be allowed to reach possibly sensitive services like RabbitMQ, Erlang, dnsmasq and similar.

Currently we allow such traffic from any IP, but restrict it only for admin/pxe L2 network (input interface) and we have a reason for that - new admin networks could be created after master node setup using Nailgun API. I'm talking about multi racks feature. So nodes from other racks also need access to the services on master node, but they have IP addresses which are not known by puppet during Fuel setup. That's why I think we should keep our current policy as is, but strongly recommend users do not allow access to hosts in admin/pxe network from the outside.

If we are talking about multi rack feature, it's deployment engineer responsibility to adjust L3 firewall rules to match 'multi rack' IP addressing specific deployment requirements.
So I strictly object to taken approach by limiting sensitive services exposure basing on L2 (interface) only , taking into account that:
- not all deployments are going to be multi rack
- it disrupts security experience badly

Solution proposal:
- adjust iptables rules according to L3 (IP) admin/PXE addressing provided during Fuel menu, making it deployment engineer responsibility to adjust rules manually for multi-rack (if intended to be used)
OR
- provide in Fuel menu additional entry (field) for providing L3 addressing for multi-rack resulting in appropriate iptables configuration
In both above cases default rule for INPUT chain should be DROP.

We just have been hit with the problem that open NTP UDP port 123 tied with public IP (from the vrouter namespace on CTRL node, Fuel 7.0) was exploited for participating in large-scale attack, generating UDP responses to spoofed "monlist" requests that claimed to be from the attack target.

Quick alternative workaround for this particular NTP exploit was (if not introducing additional/external firewall):
"disabling the monitoring function by adding "disable monitor" to your /etc/ntp.conf file; OR alternately upgrading to the latest version, which removes the "monlist" command that is used for these attacks. "

This bug is medium due to several criteria

1. It doesn't break a deployment
2. It doesn't break any component
3. It doesn't make any barriers in lifecycle management

Firstly, all controllers and computes are protected by networks with DROP policy by default

if we look precisely at iptables configuration it's easy to find that most of rules are protected by interface

Hello Andres Toomsalu, thank you for reporting this issue to us.
NTP issue was discovered by us and corrected with this commit:
https://review.openstack.org/#/c/219325/

For more references, please see original bug report:
https://bugs.launchpad.net/fuel/+bug/1488731

Hello Sergii, please let me clarify issue:
1. It is Fuel master only related (let's focus here on Fuel master node)
2. In solution proposal I tried to explain proposed approach with networks services exposure. Note that this issue report is about services exposure and please let's focus on this.
3. Of course I fully agree with reasoning mentioned by you (doesn't break deployment, component, is not LCM management related) but this issue is security related, not functionality related.
4. As this issue is security related and focused of unnecessary network exposure and risks associated with it, I'm concerned here about possible DOS attack vectors possible to exploit on Fuel master node.
5. For security threats taxonomy, industry uses CVSS rating. Fuel is no exception, for security assessments we should use CVSS as a measure of Fuel security standing.
6. According to CVSS rating this issue is at level 7.5 of CVSS base score (High) , https://www.first.org/cvss/calculator/3.0#CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H
7. Assessment input values (input vector):
- attack vector (AV): Network
- attack complexity (AC): Low
- privileges required (PR): None
- user interaction (UI): None
- scope (S): Unchanged
- confidentiality (C): None
- integrity (I): None
- availability (A): High
8. Proposed mitigation steps are focused on firewall refactoring, which would prevent unnecessary network exposure (limit scope of attack vector to adjacent networks) and limit CVSS rating to ratio of about 6.5 (medium), what I believe is acceptable for most of Fuel use cases.

As Artem already mentioned, this will break multi-rack feature. In 8.0 we've spent a lot of efforts to remove all the manual steps needed for Fuel users to run multi-rack deployments. So I personally don't like the idea of adding new manual step for multi-rack deployments - firewall configuration. Because it will look like "break something that works, describe manual fix for that". We can document this bug as known issue for 8.0 and describe security fix - it will be the same manual firewall configuration, but it won't break any existing functionality.

We can fix this issue in a fully automated way when we get rid of docker containers and thus will be able to run puppet tasks from Nailgun on Fuel master host system (in 9.0?)

ok, understand.

Fix proposed to branch: master
Review: https://review.openstack.org/297597

bvt is fine: http://jenkins-product.srt.mirantis.net:8080/view/custom_iso/job/9.0.custom.ubuntu.bvt_2/462/
vbox currently fails, investigating...

Fix proposed to branch: master
Review: https://review.openstack.org/299811

Reviewed: https://review.openstack.org/299811
Committed: https://git.openstack.org/cgit/openstack/fuel-virtualbox/commit/?id=3b40e0c28484b938b9958d72c936a95cbdf2e97e
Submitter: Jenkins
Branch: master

commit 3b40e0c28484b938b9958d72c936a95cbdf2e97e
Author: Dmitry Bilunov <email address hidden>
Date: Thu Mar 31 11:46:40 2016 +0300

    Move iptables rules to user-defined chains

    Firewall rules should be managed by Puppet, which will purge current
    firewall ruleset and replace it by the one provided by a corresponding
    manifest.

    We need to move rules introduced by product.sh to separate user-defined
    chains which are not purged by Puppet so they will be retained after the
    manifest gets applied.

    Change-Id: I57e9f58c6bad32b23b179499f0514edf5357bd31
    Partial-Bug: #1524750

Change abandoned by Sergii Golovatiuk (<email address hidden>) on branch: master
Review: https://review.openstack.org/297597

Fix proposed to branch: master
Review: https://review.openstack.org/303326

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/303464

Reviewed: https://review.openstack.org/303326
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=77339d40c341f778e5fec3701c9012c0a4f39c7e
Submitter: Jenkins
Branch: master

commit 77339d40c341f778e5fec3701c9012c0a4f39c7e
Author: Maksim Malchuk <email address hidden>
Date: Fri Apr 8 11:48:59 2016 +0300

    Set default iptables INPUT policy to DROP

    It was observed that on Fuel master node the default firewall INPUT
    policy is set to 'ACCEPT' rather than to 'DROP'.
    This leads to exposure of unnecessary services over a potentially
    untrusted networks.

    This patch updates default firewall INPUT policy to 'DROP'. Also it adds
    user-defined chains which are not purged (and thus preserved between
    iptables.pp applications).

    Change-Id: Ia9ab6d019be81aebcf5eaba25336e6f19b2c6a1a
    Partial-Bug: #1524750
    Depends-On: I57e9f58c6bad32b23b179499f0514edf5357bd31
    Signed-off-by: Maksim Malchuk <email address hidden>

Reviewed: https://review.openstack.org/303464
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f73c7fd8fb04366c0793835ac584457f0d3a53f7
Submitter: Jenkins
Branch: stable/mitaka

commit f73c7fd8fb04366c0793835ac584457f0d3a53f7
Author: Maksim Malchuk <email address hidden>
Date: Fri Apr 8 11:48:59 2016 +0300

    Set default iptables INPUT policy to DROP

    It was observed that on Fuel master node the default firewall INPUT
    policy is set to 'ACCEPT' rather than to 'DROP'.
    This leads to exposure of unnecessary services over a potentially
    untrusted networks.

    This patch updates default firewall INPUT policy to 'DROP'. Also it adds
    user-defined chains which are not purged (and thus preserved between
    iptables.pp applications).

    Also this commit contain: Bump firewall module to 1.8.0
    (cherry picked from commit 7f17a1a0e6a578a51f00e84dfd44de90b0ee0b4d)

    Change-Id: Ia9ab6d019be81aebcf5eaba25336e6f19b2c6a1a
    Partial-Bug: #1524750
    Depends-On: I57e9f58c6bad32b23b179499f0514edf5357bd31
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit 77339d40c341f778e5fec3701c9012c0a4f39c7e)

Fix proposed to branch: master
Review: https://review.openstack.org/305443

Reviewed: https://review.openstack.org/305443
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e10a88344e2de733289fcc4c1bf30c221a16e091
Submitter: Jenkins
Branch: master

commit e10a88344e2de733289fcc4c1bf30c221a16e091
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 13 21:35:36 2016 +0300

    Restore forwarding and iptables cleanup

    Restore accidentally removed forwarding back but move it outside the
    firewall module.
    Passing port to firewall is deprecated and will be removed, so change
    all rules to use dport instead.
    Firewall rules with jump attribute should contain explicit protocol
    declaration.

    Change-Id: I750f334667966299a26c305126445524de73ff2c
    Closes-Bug: #1568891
    Partial-Bug: #1524750
    Signed-off-by: Maksim Malchuk <email address hidden>

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/305739

Reviewed: https://review.openstack.org/305739
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=1b65f52f21bb52687185ff64a3d1e690a2ed951c
Submitter: Jenkins
Branch: stable/mitaka

commit 1b65f52f21bb52687185ff64a3d1e690a2ed951c
Author: Maksim Malchuk <email address hidden>
Date: Wed Apr 13 21:35:36 2016 +0300

    Restore forwarding and iptables cleanup

    Restore accidentally removed forwarding back but move it outside the
    firewall module.
    Passing port to firewall is deprecated and will be removed, so change
    all rules to use dport instead.
    Firewall rules with jump attribute should contain explicit protocol
    declaration.

    Change-Id: I750f334667966299a26c305126445524de73ff2c
    Closes-Bug: #1568891
    Partial-Bug: #1524750
    Signed-off-by: Maksim Malchuk <email address hidden>
    (cherry picked from commit e10a88344e2de733289fcc4c1bf30c221a16e091)

Checked iptable's rules.

Verified on:
cat /etc/fuel_build_id:
 467
cat /etc/fuel_build_number:
 467
cat /etc/fuel_release:
 9.0
cat /etc/fuel_openstack_version:
 mitaka-9.0