Isolating Subnets in pfSense
The pfSense firewall distribution is one of my favourite pieces of software. It is powerful and flexible, has wide adoption, and is under active development.
For the most part, the GUI for firewall rules is intuitive to use. But it has a huge problem: it makes isolating subnets unintuitive. Consider the following set of networks:
- WAN (which we won't care much about in this entry)
- LAN (which is our "home" network)
- GUEST (which is used for guest access)
- WORKSHOP (which is a lab of computers for development)
GUEST uses the OPT1 interface, and WORKSHOP uses the OPT2 interface.
I want to accomplish the following:
- Allow LAN/GUEST/WORKSHOP access to the wider internet
- Isolate the LAN/GUEST/WORKSHOP subnets from each other, so that by default no traffic will flow between them
- Have the ability to poke holes between the subnets for specific purposes
For years, I assumed that this was easy to set up. For years, I have been doing it wrong, and unless you are careful you might do so as well.
Terminology
Let's define the following aliases:
all_subnets
is a pfSense network alias, that includes the subnets for LAN, GUEST and WORKSHOP
Let's define the following isolation exceptions:
- All hosts on LAN/WORKSHOP (and GUEST, by convention) should be able
to SSH into
nethack_hosts
on GUEST (using TCP port 22) - All hosts on LAN should be able to access an NTP server
ntp_host
on WORKSHOP (which uses UDP port 123) [CHECK]
Let's assume that "subnet" is a synonym for the pfSense concept of "interface" (even though that is not strictly true)
Failed Attempt 1: Defensive Subnets
My first approach was as follows:
- Each subnet has rules that block incoming traffic from other subnets
- Exceptions are listed before the block rules
- The last rule for each subnet is an "Allow everything" rule to permit access to the wider internet
Here is what the rules for LAN would be (in pseudocode, not real pf notation):
- Pass all from "LAN Subnet" to
nethack_hosts
on port 22 - Pass all from "LAN Subnet" to
ntp_host
on port 123 - Pass all from "LAN Subnet" to "LAN Subnet"
- Block all from
all_subnets
to "LAN Subnet" - Pass all from "LAN Subnet" to any
Here are the rules for GUEST:
- Pass all from "GUEST Subnet" to "GUEST Subnet"
- Block all from
all_subnets
to "GUEST Subnet" - Pass all from "GUEST Subnet" to any
Here are the rules for WORKSHOP:
- Pass all from "WORKSHOP Subnet" to
nethack_hosts
on port 22 - Pass all from "WORKSHOP Subnet" to "WORKSHOP Subnet"
- Block all from
all_subnets
to "WORKSHOP Subnet" - Pass all from "WORKSHOP Subnet" to any
I am not actually sure whether the
- Pass all from "LAN Subnet" to "LAN Subnet"
- Pass all from "GUEST Subnet" to "GUEST Subnet"
- Pass all from "WORKSHOP Subnet" to "WORKSHOP Subnet"
are actually necessary (in this attempt or any others) but I included them to be safe.
Here are the problems with this approach:
- Any host on the LAN subnet can pass traffic freely to the GUEST or WORKSHOP networks
- Any host on the GUEST subnet can pass traffic freely to the WORKSHOP network
- Hosts on the WORKSHOP network cannot access the
nethack_hosts
on the GUEST network, despite the firewall rule in the WORKSHOP subnet.
To understand this behaviour, you need to understand what pfSense does
behind the scenes in translating rules from the nice GUI into actual
pf
firewall rules that the underlying FreeBSD system can use. Here
is my understanding of how rules are generated:
- pfSense first makes some magic rules to allow traffic in and out of the firewall
- Then it converts firewall GUI rules, tab by tab in the following order: floating rules, WAN, LAN, OPT1, OPT2, OPT3... .
- Within a tab, it converts rules in order
The quick
keyword has an effect here (especially for floating rules)
but as far as I can tell it does not help us much.
You need not take this assertion on faith: just SSH into your pfSense
box and run the pfctl -s rules
or pfctl -vv -s rules
commands.
What this means for our example is that all rules on the LAN tab are processed before any rule in the GUEST (aka OPT1) or WORKSHOP (aka OPT2) tabs. In particular, the following rule:
- Pass all from "LAN Subnet" to any
is executed before
- Block all from
all_subnets
to "GUEST Subnet"
or
- Block all from
all_subnets
to "WORKSHOP Subnet"
rules. That first rule allows traffic to go from the LAN Subnet to any other subnet, including the GUEST and WORKSHOP ones, so the later rules do not matter. Oops!
Similarly, the rule:
- Block all from
all_subnets
to "GUEST Subnet"
in the GUEST tab is processed before
- Pass all from "WORKSHOP Subnet" to
nethack_hosts
on port 22
on the WORKSHOP tab, which means that the WORKSHOP subnet will be
blocked from nethack_hosts
regardless of the second rule.
In retrospect (and once I saw what was going on with pf
rule
generation) this behaviour made perfect sense. But it is not the
behaviour I expected when I first used the GUI interface -- for some
reason I thought that pfSense would magically know that I wanted to
block traffic between subnets but still allow generalized internet
traffic.
It is possible that you could massage this particular example into working with a "defence first" approach, but I kept getting stuck, so I started investigating floating rules.
Semi-Failed Attempt 2: Floating Rules
pfSense version 2.0 introduced the idea of "floating rules" -- rules
that can apply to multiple interfaces, and which would be processed
before any of the interface-specific tabs. I thought I could use this
to poke holes in the isolated subnets (which would solve the problem
of WORKSHOP getting access to nethack_hosts
above).
The real problem with this approach is preventing LAN from accessing the other subnets willy-nilly. I came up with something that kind of worked, but it effectively required all rules to be in the "Floating" tab:
- First, put in specific exceptions to subnet isolation
- Then isolate the subnets
- Then allow all traffic to pass
The ruleset would look something like this:
- Pass all from
all_subnets
tonethack_hosts
on port 22 Pass all from "LAN Subnet" to
ntp_host
on port 123Pass all from "LAN Subnet" to "LAN Subnet"
- Pass all from "GUEST Subnet" to "GUEST Subnet"
Pass all from "WORKSHOP Subnet" to "WORKSHOP Subnet"
Block all from
all_subnets
toall_subnets
- Pass all from
all_subnets
to any
(I simplified things a little by using all_subnets
as shorthand for
individual subnets in a few places.)
This looks workable for this example, but it has a real disadvantage: effectively, all rules have to be in the "Floating" tab, which means the other tabs do not get used. This is not bad for a simple example, but it gets quite messy (and therefore quite error-prone) when you have seven interfaces with many unique exceptions per interface. Making matters worse is that the tabular display for floating rules does not indicate the set of interfaces for which the rule applies -- if I have a rule that should ONLY apply to the GUEST interface, I cannot see this until I click into the rule.
One wildcard with this approach is the "quick" keyword. I never figured out how to use it properly in this context (although I have used it in traffic shaping, to put traffic into queues).
Working (?) Attempt 3: Polite Subnets
Here is the approach that ended up working for me. No doubt it is the approach that most people come up with from the beginning, but I is dumb:
- All subnets initiate traffic to other subnets that poke holes in the subnet isolation
- Then all subnets prohibit their own traffic from entering other subnets
- Then the subnets allow traffic to the broader internet
This takes the opposite approach from Attempt 1: instead of subnets defending themselves from unwanted traffic from other interfaces, they depend on other interfaces prohibiting unwanted traffic from escaping to them. From a security standpoint this seems insane -- very few security scenarios assume that other actors (in this case, other subnets) are benevolent. But in our case all sysadmins control the behaviour of all subnets, and so should be able to coordinate the actions of those subnets together. If you somehow had a situation where different sysadmins had access to only the firewall rules of their own subnet (and could not be trusted to behave well towards other subnets) then this approach would fail.
With this approach, here is what the rules look like for LAN:
- Pass all from "LAN Subnet" to
nethack_hosts
on port 22 - Pass all from "LAN Subnet" to
ntp_host
on port 123 - Pass all from "LAN Subnet" to "LAN Subnet"
- Block all from "LAN Subnet" to
all_subnets
- Pass all from "LAN Subnet" to any
Here are the rules for GUEST:
- Pass all from "GUEST Subnet" to "GUEST Subnet"
- Block all from "GUEST Subnet" to
all_subnets
- Pass all from "GUEST Subnet" to any
Here are the rules for WORKSHOP:
- Pass all from "WORKSHOP Subnet" to
nethack_hosts
on port 22 - Pass all from "WORKSHOP Subnet" to "WORKSHOP Subnet"
- Block all from "WORKSHOP Subnet" to
all_subnets
- Pass all from "WORKSHOP Subnet" to any
It looks almost the same as Attempt 1, except that rules like:
- Block all from
all_subnets
to "LAN Subnet"
are switched to
- Block all from "LAN Subnet" to
all_subnets
This approach is weird to my brain, but (as far as I can tell) it
works. Unlike Attempt 2 it allows you to use the firewall GUI tabs in
a helpful way. It also scales well to many many subnets, just by use
of the all_subnets
alias.
It could very well be that my approach is STILL all wrong. Maybe it is catastrophically wrong! I feel stupid even posting this on the Internet, given that I am far from a pfSense expert (and I expect pfSense experts are scoffing at me now). But I believe this gets me a lot closer to the behaviour that I want than my earlier approaches, so I thought I would share.