Paul's Internet Landfill/ 2016/ Failing Over Between pfSense Boxes

Failing Over Between pfSense Boxes

At work we have several buildings. Most of our buildings use pfSense for firewalling and splitting off subnets.

Say we have two buildings, "Building-Office" and "Building-Cafe", that are physically near each other. Each of these buildings has its own Internet connection and pfSense box: pfsense-office and pfsense-cafe. Each building has a set of subnets associated with it, as follows:

Building-Office has the following subnets:

Building-Cafe has the following subnets:

As you can see, both buildings have a STAFF subnet with the same IP address range. There is a wireless bridge that connects the two buildings on the STAFF subnet. On the STAFF subnet, pfsense-office has an IP address of 192.168.100.254 and pfsense-cafe has an IP address of 192.168.100.253 .

I have the following goals:

I have a feeling that any competent network admin could set up pfSense to accomplish these goals within minutes (which is one reason I have felt intimidated about asking this question online). It took me YEARS to get something working properly, so I want to document the procedure that works for me in the hopes that other people can learn from my incompetence.

As it turns out there are a few more important considerations to our situation:

Non-Solutions

Phase 1: Failing over STAFF

Failing over STAFF over the wireless link is relatively easy. The key is to specify some new gateways in System -> Gateways of the pfSense interface:

At this point failover across the wireless for the STAFF interface should be possible. Say that the internet connection goes down at Building-Office. Then to fail over STAFF to Building-Cafe, do the following:

If for some reason you have different sets of firewall rules for the STAFF interfaces be aware that the rules for the pfsense-cafe STAFF interface will apply during failover.

Phase 2: Failing over other subnets

This is where things get tricky. The wireless link is on the STAFF network, so we need to route other traffic via that interface. Here are the broad steps:

Let's set up the failover for the LAB and WORKSHOP subnets over pfsense-cafe.

On pfsense-cafe, set up an alias called office_subnet_failover. It should consist of two networks: WORKSHOP: 10.10.10.0/24 and LAB: 10.20.0.0/24

On pfsense-cafe, go to System -> Routing -> Routes. Make a new static route with the destination network office_subnet_failover and the gateway GW_OFFICE. Make the description descriptive: "Fail over pfsense-office subnets."

On pfsense-cafe, in Firewall -> Rules -> STAFF make a firewall rule:

Be careful! If you are using subnet isolation then you want to put this rule after your isolation rules so that LAB and WORKSHOP clients cannot access STAFF resources.

If you have manual NAT, go to Firewall -> NAT -> Outbound and make a NAT rule:

If you do not have this rule then there will be no NAT for outgoing packets on LAB and WORKSHOP, and the destinations will try to return packets to your internal subnets instead of the actual IP address of pfsense-cafe.

As far as I know, this is sufficient to get failover working (in one direction). You can set up a similar set of rules on pfsense-office to failover the CAFE subnet.

You then "flip the switch" on the failover in the same way as in Phase 1: Make the appropriate gateway the default, and disable the other one.

If you are paranoid you can also disable the failover firewall rules until it is failover time, and then enable them to make failover work. But this adds additional steps to flipping the switch.