OpenLDAP N-Way Multi-Master Replication on Synology DSM6

We went through the pain a few years back of getting this working in slapd.conf on DSM5 but needed to configure recently on DSM6 (which now uses cn=config). It took a while to crack but is really simple now we know how!

Install Directory Server from the Package Center on every node

Launch Directory Server and configure the basic settings on every node;

(I would suggest checking the “Disallow anonymous binds” option under “Connection Settings”);

From the control panel, select the Domain/LDAP option and check “Enable LDAP Client” on the LDAP tab. Enter localhost as the server address, SSL for encryption and the BaseDN from the Directory Server settings screen then click Apply.

Now use JXplorer (or your LDAP tool of choice) to connect to the cn=config database (again, you will need to repeat this step for every node);

You should see something like;

Switch to Table Editor, right click olcSyncrepl and chose “Add another value”. Then you need to paste;

{1}rid=002 provider=ldap://dnsname.synology.me:12345 bindmethod=simple timeout=0 network-timeout=0 binddn="uid=root,cn=users,dc=tick,dc=home" credentials="password123" keepalive=0:0:0 starttls=no filter="(objectclass=*)" searchbase="dc=tick,dc=home" scope=sub schemachecking=off type=refreshAndPersist retry="60 +"

You will need to replace;

• the provider (depending on whether you are using a VPN, have a static IP etc)
• the binddn (you will find this on the main screen of the Directory Server app as per my earlier screenshot)
• the credentials (this is the password you configured when configuring the Directory Server earlier)
• the searchbase (you will find this on the main screen of the Directory Server app as per my earlier screenshot)

Then locate olcMirrorMode and click into the value column and select True;

If you have more than 2 nodes in your n-way multi-master replication “cluster” you will need to add an additional olcSyncrepl entry for each node (be sure to increment the {1} and 002.

That’s it (I rebooted for good measure). Now try creating a user on each node and check it appears on your other nodes.

If you experience any issues your best bet is probably checking /var/log/messages

Good luck!

Building Hosting Environment: Part 1- Hardware

I hope to be installing some equipment in a local datacenter to offer some hosting services. First item, the hardware;

• Ubiquiti Edgerouter Lite
• Dell 8024 (24x 10GbE Switch)
• Synology RS3614RPXS NAS (6x WD RED 3TB + 2x Samsung EVO 840 1TB + Intel X540-T2 10GbE NIC)
• 2x Supermicro AS-2022TG-HIBQRF (each w/ four nodes w/ 64GB RAM & 2x Opteron 6176 + Intel X540-T2 10GbE NIC)

Initially I went for a combination of the Netgear Prosafe XS708E (8x 10GbE Switch) paired with a Dell (24x 1GbE Switch) but quickly found myself running out of 10GbE ports and concerned about the lack of redundant power supplies.

Likewise, I had chosen the RS3614XS but felt the additional cost of the RP (model with redundant power supplies) was justified.

And finally the servers themselves, initially Supermicro AS1042G-LTF (single node with four sockets and single power supply) but then switching to the AS-2022TG-HIBQRF (four node, each with two sockets and shared redundant power supplies).

I’ve tried to avoid single points of failure at a component level (redundant power supplies etc) but without overkill couldn’t avoid it at device level (redundant switches, NAS etc).

Supplier wise… I got the switch from http://www.etb-tech.com/ and the NAS from http://elow.co.uk/ (both of which admittedly i had my doubts about when first placing the orders, as the prices seemed a little cheap, but the service was incredible, both dispatched same day using next day couriers). The rest from eBuyer and local suppliers.

Each device is connected to the switch using 2x10GbE LAG/LACP ports (I may go more into the configuration of this later).

New Infrastructure for a Small (Multi-Site) Business

I was recently tasked with overhauling the “network” for a local, small, not for profit. The company currently have 2 sites, with roughly a dozen desktops at each and half dozen laptops which roam between the two.

The primary requirements were to provide;

• networked file storage (preferably redundant)
• centralised user management (single sign-on and access control)
• site blocking/web filtering

If both sites had “reasonable” internet connections, I would have suggested a single server at the “central” location with a site-to-site VPN. Unfortunately the connections are~ 3MBit down, 0.3Mbit up (ADSL). This introduces a need for additional hardware (servers at every site) and a way of synchronising/replicating between the sites!

As always, everything should be designed with scalability in mind, but sticking to a tight budget.

The File Servers

My first purchase were the file servers. Many years back I used to “roll my own” with something like a HP MicroServer and Windows Home Server (or possibly FreeNAS/OpenFiler) but some years back I made the transition to a dedicated Synology appliance.

Whilst you lose some of the flexibility (being able to install any software on x86/x64 hardware like the MicroServer) you gain a huge amount of reliability and support by going with a dedicated appliance (not to mention the huge feature set and ability to run many additional applications on the Synology product line).

One of the only requirements for the file server was redundancy (so at least 2 bays to support raid 1). Wanting to stick with Synology I used their product comparison tool (https://www.synology.com/en-uk/products/compare) to make a shortlist and then after looking at the prices settled for the DiskStation DS214.

Although storage requirements were pretty small, I got a great deal on WD Green 3TB disks so bought 5 (2 for each site and 1 spare).

The Routers

Had this have been a “home solution” i’d probably have opted for something like the Asus RT-AC66U flashed with one of the open source firmwares such as OpenWRT or DD-WRT. But, needing a “business solution” I needed something, most importantly reliable (the potential sacrifice being ease of use).

On top of reliability, the primary feature requirement for the routers is site-to-site VPN. After some research I decided to give the Ubiquiti EdgeRouter Lite 3 a try. Frustratingly the ADSL connection coming in at both sites is provided by a BT HomeHub3. The HH3 doesn’t support bridge mode, and to avoid double NAT / further complications I decided to purchase 2 ADSL modems (there aren’t many to chose from… I went for the Draytek Vigor 120).

Documentation

I previously posted about some SharePoint issues i’ve been tackling, this is the medium i’ve chosen for documenting and sharing the how-to guides, configuration details and process documents. I’m yet to tackle, but may also use it for new user requests, password resets, support requests etc.

To be continued…

Similarly, I have already posted about getting OpenLDAP replication working, this was one tiny part of the project. I will be following up this post with a number specifically tackling the implementation and configuration of the new solution.

Watch this space.

Synology Directory (OpenLDAP) Replication Issues

A few weeks on since my last post (https://tickett.wordpress.com/2014/08/14/synology-directory-openldap-replication/) I have found a few bugs, fixed a few more issues and hopefully have a fully working solution.

One of issues with my previous post (that i’m not going to go into at the moment) was that I hadn’t cross-compiled openssl and cyrus-sasl2 so my version of slapd didn’t support either. I think i’ve now resolved this and you can download my latest slapd here: https://dl.dropboxusercontent.com/u/713/slapd

#1- I needed the slave to refer changes to the master

Documentation and discussion everywhere seems to suggest simply adding a line to the slave slapd.conf;

updateref ldap://192.168.10.250

Would ensure any changes were written to the master but I couldn’t get this working (even with debug enabled). The only error I could really find (from memory) was an err=49 which I believe refers to invalid credentials but i’m unsure which credentials or how this is possible.

After further research, I found that there is an alternatively openldap configuration referred to as n-way multi master. Rather than specifying a master and slave, both nodes are masters and changes are replicated both ways. This was relatively easy to setup and “just worked” (not to mention, a better solution as before it was possible the “master” server would be unreachable (if the site-to-site VPN was down) and changes would fail).

You will find config details for n-way multi master / mirrormode in my next blog post.

#2- Unable to access shares after password change (from windows/pGina) with error “Element Not Found”

This was a real curve ball. Google sent me in completely the wrong direction, but I recalled a discussion about multiple passwords being stored in the LDAP database, which led me to wonder if the userPassword wasn’t the only field needing to be updated.

A colleague stumbled across the documentation for pGina fork: http://mutonufoai.github.io/pgina/documentation/plugins/ldap.html which shows a rather more complete “Change Password” configuration for the LDAP plugin. Unfortunately pGina main doesn’t support the DES or Timestamp methods so we couldn’t configure sambaLMPassword, shadowLastChange or sambaPwdLastSet, but adding sambaNTPassword (MD4) alongside userPassword (SHA1) seems to have done the trick.

#3- Data was replicating but the users could not login

I’m not sure exactly how I figured this one out. I think I had a vague recollection of reading a discussion about passwords not replication because default permissions do not allow them to be read from the database.

I added a line in slapd.conf above the existing ACL include;

include /usr/syno/etc/openldap/acls.conf
include /usr/syno/etc/openldap/slapd-acls.conf

The contents of which;

access to attrs=userPassword,sambaLMPassword,sambaNTPassword
     by dn.base="cn=replication,cn=users,dc=example,dc=com" write

Allow the password to be read from the database by the replication user.

Synology Directory (OpenLDAP) Replication

This drove me close to insanity, but I got there eventually!

I found an old discussion on the Synology forum http://forum.synology.com/enu/viewtopic.php?f=183&t=55020 and was optimistic it’d be pretty simple. The thread talks about compiling a later version of OpenLDAP from source, but the version included (in DSM5.0) is later than that discussed;

file-20> slapd -VV
@(#) $OpenLDAP: slapd 2.4.34 (Feb 27 2014 03:17:07) $
root@build3:/source/openldap-2.4.x/servers/slapd

I tried configuring my provider and consumer using the example and referring to http://www.openldap.org/doc/admin23/syncrepl.html but wasn’t getting anywhere (after changing slapd.conf I would disable and re-enable the LDAP server through the web ui). I was getting an error “Permission denied. Please contact the server administrator.” and an entry in /var/log/messages;

file-20> tail /var/log/messages
Aug 14 21:51:59 file-20 ldap.cgi: ldap_server_default_add.c:146 add [admin] to [cn=Directory Operators,cn=groups,dc=example,dc=com] failed, ldap_insufficient_access (53)

Oddly the slapd process continues to run but no replication is taking place. I believed the error might be because the admin account is locked in some way and wont allow any modification. I tried adding a filter;

filter="(!cn=admin)"

This prevented the error message popping up and the error in /var/log/messages but still no replication was taking place.

I imagine it would have been a trivial task on a standard Linux distribution but it seems OpenLDAP has been compiled in a manner which does not allow debug;

file-20> slapd -d 1
must compile with LDAP_DEBUG for debugging

So there’s no real feedback as to what is (or isn’t) working.

After blindly fumbling around for hours I decided to try and compile myself so I could debug. This itself was a mammoth chore!

I wanted to stick with the same version currently running on DSM5.0 so started with the source for 2.4.34 from http://www.openldap.org/software/download/OpenLDAP/openldap-release/

In order to cross compile I followed the Synology 3rd-Party Package Developers guide; http://www.synology.com/en-uk/support/third_party_app_int. I had a spare ubuntu machine I could use for compiling… I needed the DSM5.0 toolchain from http://sourceforge.net/projects/dsgpl/files/DSM%205.0%20Tool%20Chains/ as i’m using the DS214 which apparently has a marvell amanda xp processor. And extracted the archive;

tar zxpf gcc464_glibc215_hard_armada-GPL.tgz –C /usr/local/

Then Berkeley DB 5.1.25 from http://pkgs.fedoraproject.org/repo/pkgs/libdb/db-5.1.25.tar.gz/06656429bfc1abb6c0498eaeff70cd04/

tar xvfdb-5.1.25.tar.gz
cd db-5.1.25
cd build_unix
export CC=/usr/local/arm-marvell-linux-gnueabi/bin/arm-marvell-linux-gnueabi-gcc
export LD=/usr/local/arm-marvell-linux-gnueabi/bin/arm-marvell-linux-gnueabi-ld
export RANLIB=/usr/local/arm-marvell-linux-gnueabi/bin/arm-marvell-linux-gnueabi-ranlib
export CFLAGS="-I/usr/local/arm-marvell-linux-gnueabi/arm-marvell-linux-gnueabi/libc/include -mhard-float -mfpu=vfpv3-d16"
export LDFLAGS="-L/usr/local/arm-marvell-linux-gnueabi/arm-marvell-linux-gnueabi/libc/lib"
../dist/configure --host=armle-unknown-linux --target=armle-unknown-linux --build=i686-pc-linux --prefix=/usr/local 

I also had to install;

sudo apt-get install lib32z1

Now I was able to configure OpenLDAP;

export LDFLAGS="-L/usr/local/lib -L/usr/local/BerkeleyDB.5.1/lib -R/usr/local/lib -R/usr/local/BerkeleyDB.5.1/lib"
export LD_LIBRARY_PATH=/usr/local/BerkeleyDB.5.1/lib
export LD_RUN_PATH=/usr/local/BerkeleyDB.5.1/lib
export CPPFLAGS="-I/usr/local/BerkeleyDB.5.1/include"
./configure --host=armle-unknown-linux --target=armle-unknown-linux --build=i686-pc-linux --prefix=/usr/local --with-yielding-select=no --enable-crypt

But when I tried to;

make depend
make

I received an error; cross compile openldap error: undefined reference to `lutil_memcmp’ – http://zhuqy.wordpress.com/2010/04/22/cross-compile-openldap-error-undefined-reference-to-lutil_memcmp/ put me straight- I just had to comment out a line from include/portable.h;

//#define NEED_MEMCMP_REPLACEMENT 1

make was now successful and I moved my newly compiled slapd to the synology diskstation, chown’d & chmod’d it,  and tested debug… we see an instant result;

file-20> chown root:root slapd.me
file-20> chmod 755 slapd.me
file-20> slapd.me -d 1
ldap_url_parse_ext(ldap://localhost/)
ldap_init: trying /usr/local/etc/openldap/ldap.conf
ldap_init: HOME env is /root
ldap_init: trying /root/ldaprc

Now I disabled the directory server in the web ui and instead ran my new version from the commandline with debug 1;

./slapd.me -d 1 -f /usr/syno/etc/openldap/slapd.conf

It failed with an error referring to;

password-hash {CRYPT}

Turns out I had to recompile slapd with –enable-crypt. I copied the newly compiled slapd over, ran again with -d 1 and now I could see it failing with error relating to an invalid filter;

filter="(!cn=admin)"

So I removed this… Try again, now;

ldap_sasl_bind_s failed

I think that sent me in the wrong direction (I thought it was an ssl/tls/authentication issue) and I spent hours messing with certificates, unsupported tls configuration parameters etc but got nowhere. Eventually I determined this error essentially means “can’t connect”. Eventually I tried without ssl and as if by magic everything sprung to life!

Here are the lines I added to the default slapd.conf on the provider;

index entryCSN eq
index entryUUID eq

overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 10

And the consumer;

index entryCSN eq
index entryUUID eq

syncrepl rid=20
 provider=ldap://192.168.10.250
 type=refreshAndPersist
 interval=00:00:10:00
 searchbase="dc=example,dc=com"
 bindmethod=simple
 binddn="uid=admin,cn=users,dc=example,dc=com"
 credentials=password
 scope=sub
 retry="60 +"

If you want to download my compiled version of slapd you can find it here; https://www.dropbox.com/s/sfb06uo0leqxqq9/slapd

I hope this will help you!

New Lab / NAS

Far too long since the last post. Let’s hope this will be the start of them picking back up again!

I have been experiencing some performance issues and need to have a bit of a re-shuffle of the servers/network (my vCenter appliance has stopped working, SQL is being slow etc). I have some production stuff running and don’t want to take everything offline for long so decided to build a new environment then migrate stuff.

I wont be changing much; 

Old NAS; Synology DiskStation 1812+ w/
-4x 3TB WD Green in Synology Hybrid Raid (SHR) : Main data store for Movies, PVR Recordings, ISOs, Photos etc (CIFS & NFS)
-2x 256GB OCZ Vertex4 SSD in RAID0 : Virtual machine storage (NFS)
-2x1gbit LACP to switch
Old ESXi Host; SuperMicro X8SIL-F w/ Xeon X3470 & 16GB RAM running VMWare ESXi v5.1
Old switch; Linksys SRW2024W

New NAS; Snology DiskStation 1813+ w/
-3x 4TB WD Red in Synology Hybrid Raid (SHR) : Main data store for Movies, PVR Recordings, ISOs, Photos etc (CIFS & NFS)
-3/4?x 250GB Samsung EVO 840 SSD in RAID0? : Virtual machine storage (NFS/iSCSI?)
-3x1gbit LACP to switch dedicated to main data store
-1gbit to switch dedicated to VM storage
New ESXi Host; SuperMicro X8SIL-F w/ Xeon X3470 & 32GB RAM running VMWare ESXi v5.5
New switch; Cisco SG200-26 (separate vm storage traffic on it’s own VLAN/subnet)

You’ll notice a bunch of questions marks around the new Virtual machine storage volume. I’m currently debating which disk configuration to use and which storage protocol. I’ve always used NFS as it seems much simpler but understood iSCSI to be the better option (especially with the Synology supporting VAAI hardware acceleration). But despite this, i’ve been reading that NFS seems to outperform iSCSI.

Additionally, if I go iSCSI I will try using 2x1gbit ports and enabling multipathing / round-robin. If I go down the NFS route I don’t think LACP will provide any benefit as the IP hash from a single ESXi host to the single DiskStation will always use the same link?

I have 4 of the EVO SSD so am initially creating a 2 disk RAID0 volume using NFS and an identical volume using iSCSI. I can then try running some like for like comparisons/benchmarks to determine which configuration to use going forward.

I will provide an update shortly.