Netgate 4200

This page assumes you already know your way around PfSense.PfSense in general and focuses on how we configure and use it for events. If you are new to PfSense, please reference PfSense's online documentation: https://docs.netgate.com/pfsense/en/latest/ 

Markey's Stock Config Features

Here is a broad overview of settings and configurations that are applied in our stock config.

Interfaces

Port 1 - WAN 1

Port 1 is configured as the primary WAN port and consumes DHCP. If you need to set static WAN information, first go to System > Routing -> Gateways to configure your static gateway, then go to Interfaces -> PORT1WAN to configure your interface IP and gateway.

Port 2 - WAN 2

Port 2 is the failover WAN port and also consumes DHCP. It is possible to bond the two WAN ports together for aggregated throughput, but we have opted not to do this so that the network setup is as "normal" as possible as much as possible.

Port 3 - LAN

Port 3 is the primary LAN port for the main event network. It is configured at 10.3.0.1/22 and serves DHCP.

Port 4 - Management

Port 3 is intended to be a "rescue" interface, a backup way to get into the gateway in the event that Port 3 is somehow misconfigured. Port 4 is configured at 192.168.1.1/24. This port does not serve DHCP, as the DHCP server could easily be the thing that broke the main LAN. You will need to configure your device with another static IP in this subnet in order to access the gateway via this interface. Also note that this port will not grant internet access; firewall rules only allow management traffic (HTTPS from 192.168.1.1/24 to 192.168.1.1).

The console port is unused.

Access

As indicated on the login screen, you can find login information in Bitwarden or on a card that comes in the case with the unit. 

Services

DHCP

Our Netgate is configured to serve DHCP for the primary LAN. The LAN network is 10.3.0.0/22, so host IPs can range from 10.3.0.1 to 10.3.3.254. The DHCP server in PfSense is configured only to serve IPs from 10.3.1.0 to 10.3.3.254, leaving 10.3.0.2 to 10.3.0.254 available for static hosts.

Remember that 10.3.0.3 through 10.3.0.10 are used by Gather Live server clusters. It is recommended not to use these IPs.

DNS

DNS is configured in a two-tier setup. A local DNS server answers queries for internal hosts; records are added automatically to match DHCP leases. The internal domain is lan.markeys.net.

Perhaps at some point in the future, we standardize lan.markeys.net and start creating real certificates for some of our internal-use devices.

There are two upstream public DNS servers configured. PfSense will send queries to both and gradually build a reputation for each over time, preferring the more performant one. Both upstream DNS servers are configured to use DoT in order to prevent MITM DNS. Both WAN interfaces are configured to ignore DNS servers provided by any DHCP lease and will use the two statically configured DoT servers instead.

The two configured upstream DoT servers are Cloudflare Security and Quad9, both with malware blocking features. 

NAT rules are configured to redirect any plain DNS queries from LAN to PfSense transparently. No unsecured DNS queries ever exit the WAN interface.

Yes, we are simultaneously preventing upstream MITM while performing MITM ourselves for DNS. Call it "being a good steward of public network resources." It's also a necessary component of how QOS works.

Any queries for FQDNs under services.speakerreadyroom.co are forwarded to 10.3.0.10, the VIP for a Gather Live server cluster.

NTP

PfSense is configured to respond to NTP requests, and NAT rules redirect any outgoing NTP requests to PfSense. This helps to keep all LAN hosts consistent.

QOS

There are a couple of layers to how QOS is accomplished, and all have to be in place in order for it to work as expected.

Firewall Aliases

An alias makes it easy to create a predefined "variable" or group that can be used in firewall rules. This way, you can create several rules based on a single alias, and then you can change the alias to change the behavior of all those rules simultaneously. PfSense features three different types of aliases, and we use all of them.

In order to keep management easy, aliases are defined for Vimeo's streaming endpoint, IPs used by Zoom, and ports used by Zoom.

DNS

PfSense uses a Layer 4 firewall; it is not inherently aware of DNS. In order to identify traffic going to Vimeo, the firewall uses whatever IP is cached by PfSense's local DNS server. Vimeo uses a CDN, so their IPs can change. In order to make sure Vimeo's streaming FQDN means the same thing to LAN hosts as it does to the firewall, those hosts must be using the PfSense gateway as their DNS server. 

PfSense's DHCP server does offer its own IP as the DNS server, so this should just work out of the box. The NAT rule that enforces basic DNS to be handled by PfSense also helps prevent any stragglers.

Queues

QOS is applied on the way out of an interface, so all the queues are on the WAN interfaces. When a queue is assigned to a packet by a firewall rule, it's just a string indicating the name of the queue. As long as queues with the same names are set up on both WAN interfaces, queueing will continue to work in a failover event.

Each interface has five queues that are labeled "highest", "high", "medium" (the default), "low", and "lowest". Each queue is assigned a priority. The queuing function is PRIQ, which has a very simple logic: higher-priority packets go first. That's it. Use this power wisely. If something in a higher queue can use a lot of bandwidth (e.g. a large file upload), then nothing in lower queues will get through at all.

In our stock config, Vimeo streams and Zoom Meeting/Webinar traffic are assigned to the "highest" queue. The other queues (besides the default "medium") aren't doing anything out of the box, but they are available for admins to use as they desire.

Here's a rule of thumb for how to use the queues:

"Highest" is for traffic that is show-critical and time-sensitive, such as a livestream. "High" is for traffic that is show-critical but not as time-sensitive as streaming video. For example, this might be an event website/app or Gather Online. "Medium" is the default queue. Everything goes here unless a firewall rule tells it to go elsewhere. "Low" is for background traffic that you don't want to impede anything else. This is a good place for traffic such as Dropbox or Sharepoint. "Lowest" is where you put stuff that you don't care about. This queue has the lowest possible priority; it absolutely will not obstruct any other traffic, though it can get through just fine if other queues aren't being used.

All queues except "highest" and "high" have RED enabled, which helps prevent any one connection from taking over available bandwidth.

Firewall Rules

Queues are applied on the way out of an interface; rules are applied on the way in. The primary LAN interface is where all of the firewall rules live that control QOS behavior. Our standard config has two QOS rules: one to send Vimeo streams to the "highest" queue, and one to send Zoom Meeting/Webinar traffic to the "highest" queue. Healthy use of firewall aliases helps to fit more semantic or logical information into fewer rules, making the result easier to understand and manage.

One QOS rule looks like this:

On the primary LAN interface, in order to catch traffic in the "upload" direction. Set Action to Pass. Define the protocol, source, and destination to match the traffic you want to queue. (Firewall aliases are your friends here.) Under Advanced Options, select the Queue from the dropdown.

Things to Do on Site

Gateway Setup

Many venues give you static IP information for your WAN rather than handing out DHCP. When this happens, you want to do the following steps in this order:

Go to System -> Routing -> Gateway Groups and delete DHCP_Failover. Go to System -> Routing -> Gateways and delete the DHCP gateways. Create new gateway(s) using the information given to you by the venue. If applicable, create a new gateway group and add your static gateways to it. Go to Interfaces -> WAN1, configure as a static interface with the appropriate IP, and select the appropriate gateway. If applicable, do the same for WAN2.

Now, your internet connection should be up.

Under certain conditions, if you are using a dual WAN setup, you may need to take some additional steps to get gateway monitoring working. Another wiki page is under construction to lay out the details.

Adding QOS Rules

The basics are done already, but you may have additional things you want to shape based on the needs of the event. First, there are a few reasons why we only apply traffic shaping in the upload direction.

There's no congestion to alleviate. As long as the LAN interface connection speed is faster than the internet connection (which is "always" in practice), all traffic that reaches the firewall will pass straight through as it is received. Any bottlenecks are upstream of us and not controlled by us. PfSense is a stateful firewall. All firewall rules and operations are applied not really on the packet level, but on the connection level. (This is also why changes to firewall rules don't affect connections that are already established.) When you apply a rule to a connection, return traffic is handled the same way. Any rules that would be made in the download direction could only apply to connections that are initiated from the outside. We don't control packet flow that gets to us. If a website is sending packets to our firewall, we can't stop our firewall from receiving them. Even if we drop them right away and don't respond, they still use bandwidth. We could apply speed limits, but that doesn't accomplish anything except forcing an onslaught of packets to sit in our firewall's memory for a while before being forwarded. Better to let it pass straight through so we can spend our hardware resources of what we care about. Upload is what we care about. The primary use case of QOS in for us is to ensure the deliverability of video streams to Vimeo. That's upload.

Do you want to identify traffic by source, by destination, or both?

If by source...

Firewall IP aliases are your friend. Create an IP alias, give it a helpful name (such as vmix_laptops), and add the IPs for the devices you want to prioritize. This way, it's easy to change that IP list later.

If by destination...

This part gets tricky. Each website is unique. Vimeo streams are easy because they give us the RTMPS URL when we create a live event. Zoom also makes it easy by publishing the IPs that they use and offering a plain text file that is made for firewall systems like PfSense. Microsoft also publishes a list, but they don't serve it in a text file format that PfSense likes; they only serve the info as a REST API that returns JSON, which isn't easily consumable by PfSense.

Unfortunately, the best guidance I can give here is to start by searching the internet for something like "Dropbox firewall rules" and just see what you can find. 🤷

Lists that are provided as a text file can be imported as a URL Alias in PfSense. You can use the zoom_meeting_webinar_ips alias as an example.

Choose a Queue

When building a firewall rule, expand the Advanced Options section and scroll to the bottom to see the dropdown for queue selection.

Order Matters.

Remember that firewall rules are evaluated from top to bottom. The first rule that matches a given packet is the one that applies.