Tag Archive: ERL


I hope to be installing some equipment in a local datacenter to offer some hosting services. First item, the hardware;

  • Ubiquiti Edgerouter Lite
  • Dell 8024 (24x 10GbE Switch)
  • Synology RS3614RPXS NAS (6x WD RED 3TB + 2x Samsung EVO 840 1TB + Intel X540-T2 10GbE NIC)
  • 2x Supermicro AS-2022TG-HIBQRF (each w/ four nodes w/ 64GB RAM & 2x Opteron 6176 + Intel X540-T2 10GbE NIC)

Initially I went for a combination of the Netgear Prosafe XS708E (8x 10GbE Switch) paired with a Dell (24x 1GbE Switch) but quickly found myself running out of 10GbE ports and concerned about the lack of redundant power supplies.

Likewise, I had chosen the RS3614XS but felt the additional cost of the RP (model with redundant power supplies) was justified.

And finally the servers themselves, initially Supermicro AS1042G-LTF (single node with four sockets and single power supply) but then switching to the AS-2022TG-HIBQRF (four node, each with two sockets and shared redundant power supplies).

I’ve tried to avoid single points of failure at a component level (redundant power supplies etc) but without overkill couldn’t avoid it at device level (redundant switches, NAS etc).

Supplier wise… I got the switch from http://www.etb-tech.com/ and the NAS from http://elow.co.uk/ (both of which admittedly i had my doubts about when first placing the orders, as the prices seemed a little cheap, but the service was incredible, both dispatched same day using next day couriers). The rest from eBuyer and local suppliers.

Each device is connected to the switch using 2x10GbE LAG/LACP ports (I may go more into the configuration of this later).

Oddly, it’s quite hard to find Ubiquiti hardware in the UK, but i’ve previously sourced equipment from an eBay distributor (http://stores.ebay.co.uk/ubntshop/) and they were able to provide the best price for the ERL.

I fired the first EdgeRouter up and starting getting to know the webUI. It didn’t take long, but seemed very basic. Even for my relatively simple requirements i’d need to get to know the CLI. The official Ubiquiti Edgemax support forum (https://community.ubnt.com/t5/EdgeMAX/bd-p/EdgeMAX) was a great place to start.

An important thing to note is that the Edgemax operating system is based on Vyatta, so you if you struggle to find an Edgemax specific solution to a problem you may be able to find a Vyatta solution which will work on your Ubiquiti hardware.

IP Addresses

One of the first decisions was an IP addressing scheme. I decided to use 192.168.x.y;

  • where x represents the site (in increments of 10 to allow for future splitting) and
  • y will be the same at every site for the router & file server

The DHCP range will be from .1 to .199

Firmware Upgrade

I should have done this a little earlier, but when configuring the system parameters I was reminded to check for a firmware upgrade and found the shipped unit was running a pretty outdated v1.1 (the current at the time of posting is v1.5). So I went ahead and upgraded.

Default Configuration (WAN + 2LAN)

The new firmware has a great wizard to get you started. I chose the WAN + 2LAN setup and was immediately up and running with the router providing internet connectivity to the LAN. However, at this point in time double NAT is occurring as the internet connection is provided by a BT HomeHub3 (which doesn’t support bridge mode).

ADSL Modem

To avoid the double NAT scenario it was necessary to purchase an ADSL modem. There don’t appear to be many to chose from, I opted for the Draytek Vigor 120. Absolutely no configuration of the modem was required, I simply plugged it in and set the ERL WAN connection to use PPoE with login credentials;

username: bthomehub@btbroadband.com
password: BT

…and voila!

VPN

Initially during testing I placed both Edgerouters side by side, set static IP addresses (8.10.0.1 and 8.20.0.1) on connected them with an ethernet cable. Unfortunately, I was unable to get an IPSec tunnel established using the WebUI, but after looking at some sample configs on the forum I was able to get it working using the CLI.

I had to then modify 3 elements to get it working on-site;

  • the peer to use the dynamic hostname
  • the local-ip to use 0.0.0.0
  • the interface to use pppoe0
vpn {
    ipsec {
        auto-firewall-nat-exclude enable
        disable-uniqreqids
        esp-group FOO0 {
            compression disable
            lifetime 3600
            mode tunnel
            pfs enable
            proposal 1 {
                encryption aes128
                hash sha1
            }
        }
        ike-group FOO0 {
            dead-peer-detection {
                action restart
                interval 15
                timeout 30
            }
            lifetime 28800
            proposal 1 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
        ipsec-interfaces {
            interface pppoe0
        }
        nat-traversal enable
        site-to-site {
            peer dynamic-hostname.com {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret secret
                }
                connection-type initiate
                default-esp-group FOO0
                ike-group FOO0
                local-ip 0.0.0.0
                tunnel 1 {
                    allow-nat-networks disable
                    allow-public-networks disable
                    esp-group FOO0
                    local {
                        subnet 192.168.20.0/24
                    }
                    remote {
                        subnet 192.168.10.0/24
                    }
                }
            }
        }
    }
}

This was working well but if either internet connection dropped or router got rebooted the VPN wouldn’t automatically come back up. Supposedly dead-peer-detection should take care of this, but it doesn’t appear to be working. I decided to create a simple workaround using a cron script;

#!/bin/bash
run=/opt/vyatta/bin/vyatta-op-cmd-wrapper
$run show vpn ipsec sa | grep "up"
if [ $? == 1 ]
then
 $run restart vpn
fi

The following command creates a cron job to run the script every 5 minutes;

set system task-scheduler task vpn_monitor executable path /config/scripts/vpn_monitor.sh
set system task-scheduler task vpn_monitor interval 5m

By placing the script in /config/scripts you ensure it remains after a firmware upgrade and is included in configuration backups.

Static-Host-Mapping

We want to block a few websites (namely facebook) and rather than overcomplicating things with url-filtering / squidguard, we’ve simply set a few static host mappings;

set static-host-mapping host-name facebook.com inet 127.0.0.1

We also set a static host map for file server at the other site (as the DNS server on the local router doesn’t have any knowledge of hostnames/ip addresses serviced by the other site). Maybe at a later date I will try and find out if I can forward DNS requests to the other site before going out to the internet?

Backup

Every time I make a configuration change I download a config backup.

On one occasion the backup failed to download and the WebUI became unresponsive (rebooting the router fixed things, but the backup still wouldn’t download). I later discovered this was due to size of the /config folder after installing squidguard and downloading the category database. As I wasn’t going to be using this initially I simply removed it.

%d bloggers like this: