Tag Archive: ESXi


Following on from: https://tickett.wordpress.com/2014/11/24/building-hosting-environment-part-1-hardware/

  • Configure IPMI (either use a static IP or setup a static DHCP lease)
  • Tweak the bios (ensure options are optimised for performance rather than to minimise noise etc)
  • Add DNS* entries for your IPMI and ESX Management Interfaces
  • Install ESXi (I did everything without the need to even plug a monitor/keyboard in, IPMI is a life saver)
  • Configure your management interfaces (use the IP addresses you previously configured in DNS, and the domain name you previously selected)

Now you can login with the vSphere client and configure a few more items;

  • NTP (on the Configuration tab under Software, Time Configuration)
  • Add your datastore (i’m using NFS, so I had to add a VMKernel interface first)

Until we have our vCenter server up and running we will stick to a single NIC.

*If you don’t yet have a device which provides DNS (router), you can add entries to your hosts file for now.

*Choosing a domain name; I’ve always gone with something.local or something.home in the past, but suffered as a result. I did a little research and found some articles suggesting best practice is to use a subdomain of an internet facing domain you own http://www.mdmarra.com/2012/11/why-you-shouldnt-use-local-in-your.html. So, say you own microsoft.com, your internal domain name may be ad.microsoft.com. You configure the NETBIOS name to be whatever you like, this will be used when you logon using NETBIOS\User rather than user@ad.microsoft.com.

New Lab / NAS

Far too long since the last post. Let’s hope this will be the start of them picking back up again!

I have been experiencing some performance issues and need to have a bit of a re-shuffle of the servers/network (my vCenter appliance has stopped working, SQL is being slow etc). I have some production stuff running and don’t want to take everything offline for long so decided to build a new environment then migrate stuff.

I wont be changing much; 

Old NAS; Synology DiskStation 1812+ w/
-4x 3TB WD Green in Synology Hybrid Raid (SHR) : Main data store for Movies, PVR Recordings, ISOs, Photos etc (CIFS & NFS)
-2x 256GB OCZ Vertex4 SSD in RAID0 : Virtual machine storage (NFS)
-2x1gbit LACP to switch
Old ESXi Host; SuperMicro X8SIL-F w/ Xeon X3470 & 16GB RAM running VMWare ESXi v5.1
Old switch; Linksys SRW2024W

New NAS; Snology DiskStation 1813+ w/
-3x 4TB WD Red in Synology Hybrid Raid (SHR) : Main data store for Movies, PVR Recordings, ISOs, Photos etc (CIFS & NFS)
-3/4?x 250GB Samsung EVO 840 SSD in RAID0? : Virtual machine storage (NFS/iSCSI?)
-3x1gbit LACP to switch dedicated to main data store
-1gbit to switch dedicated to VM storage
New ESXi Host; SuperMicro X8SIL-F w/ Xeon X3470 & 32GB RAM running VMWare ESXi v5.5
New switch; Cisco SG200-26 (separate vm storage traffic on it’s own VLAN/subnet)

You’ll notice a bunch of questions marks around the new Virtual machine storage volume. I’m currently debating which disk configuration to use and which storage protocol. I’ve always used NFS as it seems much simpler but understood iSCSI to be the better option (especially with the Synology supporting VAAI hardware acceleration). But despite this, i’ve been reading that NFS seems to outperform iSCSI.

Additionally, if I go iSCSI I will try using 2x1gbit ports and enabling multipathing / round-robin. If I go down the NFS route I don’t think LACP will provide any benefit as the IP hash from a single ESXi host to the single DiskStation will always use the same link?

I have 4 of the EVO SSD so am initially creating a 2 disk RAID0 volume using NFS and an identical volume using iSCSI. I can then try running some like for like comparisons/benchmarks to determine which configuration to use going forward.

I will provide an update shortly.

I powered up a new ESX host and enabled passthrough. This got me a bit further:

I thought it had hung here but patience paid off and the installer booted:

Damn- the keyboard/mouse doesn’t work (the pointer was moving around but I couldn’t click anything). Again- patience paid off and eventually it automatically moved on to the next step:

No target disks were listed when I chose to "Reinstall OS X". I went in to Disk Utility to investigate:

Damn. I tried using both IDE and SCSI disks in ESX but neither appear.

No doubt I’ll try some other ideas at a later date. I have a feeling I need to replace a .kext file on the install disc.

L

No success unfortunately- although I’m not overly fussed about getting it to work- just fancied a try.

ESX 5:

VirtulBox:

Again, getting stuck on "Still waiting for root device".

This was mounting the iso or trying client dvd drive. I can’t take ESX offline at the moment to reboot and enable USB passthrough.

If I get an opportunity to bring my other ESX host up sometime I’ll give it another go.

L

More problems

Until a few months back I was a member of Experts-Exchange. I’ve found the site a great tool for those little problems you know countless others have encountered and could solve for me in a matter of seconds…. Anyway- this year I was thinking twice about renewing when I re-discovered Stack Exchange a very similar tool but this time at no cost. Stack Exchange has been invaluable in assisting me with a number of recent problems…

SUMIF with multiple criteria in Excel 2003

What I wanted to do seemed relatively straightforward and google threw up a lot of results but the majority of recommended solutions were for Excel 2007 and above. The remaining solutions simply didn’t seem to work?

Here’s the question/answer on Stack Overflow

In the below screenshot I was trying to sum all income for org_code (row 18) and month (column A).

It turned out I was making a few fundamental mistakes:

  • org_code is being held as a string in data table (the left-hand alignment helped Jesse spot this and point me in the right direction!
  • none of the possible solutions seem to like named ranges or evaluating entire columns (A1:A1000 works fine but A:A does not)

I’d like to find some more time to look at this (though unlikely I will). I should also have been able to use SUM and IF in an array formula too.

The working solution looks "something like"

=SUMPRODUCT(D1:D8,--(A1:A8=OrgRef),--(C1:C8=MonthRef))
Thanks Jesse!

ESXi problems

1) ESXi 5.0 relapsing to an old configuration on reboot

I’m still unsure what caused this but I did find that volumes weren’t mounting as expecting during the hypervisor boot process. Choosing recovery mode (shift-r) option during boot successfully solved the issue however.

2) Unable to power on ESXi 5.0 host from vSphere

Still a bit of a puzzle but the primary reason appears to’ve been that the host had to not only be managed to vCenter, but also had to be part of a "cluster". Once I created a cluster and moved the host into it I was able to place the physical machine into standby mode. Unfortunately, I still don’t appear to be able to power the host from an off state. If I make any further developments I will try and update this post.

Cat5e cable with cat6 connectors

There appears to be a lot of contradicting information online about the back-ward compatibility of cat6 connectors but from this experience I would have to rule that cat6 connectors are NOT backward compatible with cat5e cable. Question on Server Fault.

Let’s hope I continue to gain a lot from these sites!

L

I wanted to deploy a new OSX Lion 10.7.2 virtual machine this morning so thought I best first run the latest unlocker from : http://www.insanelymac.com/forum/index.php?showtopic=267296

I downloaded the unlocker, unzipped onto one of my datastores and tried to run it from one of my ESXi hosts:

/vmfs/volumes/9fe35ffc-47004197/unlock/esxi # ./install.sh 
VMware ESXi 5.0 Unlocker 1.0.1
==============================
Patching files...
Setting permissions...
Patching bin/vmx
open failed Permission denied

Damn- the error was coming from the Unlocker.ESXi application so I couldn’t peak at the code to see which file(s) were causing the permissions error. However, as we’re passing the newly patched files from bin/ I thought it was safe to assume these were the cause of the problem. I chmod 777 bin/* then ran ./Unlock.ESXi bin/ and the permission error had gone :)

I modified install.sh and ran again:

/vmfs/volumes/9fe35ffc-47004197/unlock/esxi # ./install.sh 
VMware ESXi 5.0 Unlocker 1.0.1
==============================
Patching files...
Patching bin/vmx
Patching bin/vmx-debug
Patching bin/vmx-stats
Setting permissions...
Creating darwin.tgz...
bin/
bin/vmx
bin/vmx-stats
bin/vmx-debug
Adding darwin.tgz to boot.cfg...
Acquiring lock /tmp/bootbank.lck
Copying darwin.tgz to /bootbank/darwin.tgz
Editing /bootbank/boot.cfg to add module darwin.tgz
sed: /bootbank/boot.cfg: No such file or directory
Editing /bootbank/boot.cfg failed: 1
Please now reboot the host system!

Doh! I’m not sure whether the missing boot.cfg is worrying or normal- but I went ahead and created an empty placeholder touch /bootbank/boot/cfg then ran the unlocker again:

/vmfs/volumes/9fe35ffc-47004197/unlock/esxi # ./install.sh 
VMware ESXi 5.0 Unlocker 1.0.1
==============================
Patching files...
Patching bin/vmx
Patching bin/vmx-debug
Patching bin/vmx-stats
Setting permissions...
Creating darwin.tgz...
bin/
bin/vmx
bin/vmx-debug
bin/vmx-stats
Adding darwin.tgz to boot.cfg...
Acquiring lock /tmp/bootbank.lck
Copying darwin.tgz to /bootbank/darwin.tgz
Editing /bootbank/boot.cfg to add module darwin.tgz
Please now reboot the host system!

Perfect :) I added the permissions fix and missing boot.cfg fix to the install.sh script- see the revised script below:

#!/bin/sh
set -e

echo VMware ESXi 5.0 Unlocker 1.0.2T
echo Modified by tickett.wordpress.com
echo =================================

# Ensure we only use unmodified commands
export PATH=/bin:/sbin:/usr/bin:/usr/sbin

# Make sure working files are removed
rm -f darwin.tgz
rm -rf ./bin

# Patch the vmx files and place into bin folder
echo Patching files...
rm -rf ./bin
mkdir bin
cp /bin/vmx bin/
cp /bin/vmx-debug bin/
cp /bin/vmx-stats bin/
## tickett: added to fix open failed Permission denied
chmod 777 bin/*
##
./Unlocker.ESXi bin/

# Ensure correct permissions are set
echo Setting permissions...
chmod 4555 bin/*

# Create the tar file for ESXi kernel
echo Creating darwin.tgz...
tar -czvf darwin.tgz bin/

# Add entry to the boot configuration file
echo Adding darwin.tgz to boot.cfg...
## tickett: added to fix error when boot.cfg missing
touch /bootbank/boot.cfg
##
BootModuleConfig.sh --add=darwin.tgz --verbose

# Clean up
rm -f darwin.tgz
rm -rf ./bin

echo Please now reboot the host system!

Or to download click : http://tickett.net/downloads/esxi_unlocker_patched_1.0.2T.zip (note this is just the patched install.sh- you will need to download the rest of the files from http://www.insanelymac.com/forum/index.php?showtopic=267296)

Thanks to Donk and Zenith432 for making Mac OS X on ESXi possible without Apple hardware!

L

Some problems

Wireless network not operating at 802.11n 130 Mbps speed (only running at 802.11g 54 Mbps): I won’t go into too much detail- but the resolution was to change the security from WPA to WPA2 (seems crazy to me that the web interface doesn’t make this incompatibility clear).

ESXi / NFS problems: My virtual machines kept dieing and there were a lot of NFS events in vSphere "Lost connection to server x mount point y mounted as z". This must be related to the network maintenance over the last few weeks but I’m not quite sure why. I ended up fitting the new HP NC360T dual-port gigabit NICs to both HP Microservers then migrating the datastore to FreeNAS. Since then *fingers crossed, touch wood etc* I’ve had no problems!

I want to talk a bit about LACP but that’ll have to wait for another time:

L

Servers Re-Racked

After running power out to the garage it was time to move the servers out and re-rack them. It all went pretty well…

Here’s the mess at the back of the rack before I stripped it out:

Once all the cabling was stripped out:

Testing the 2U cable dump panel for the power cables:

And covered up:

Seems to work nicely but I only have one at the moment and I need that for the network cables:

Power cables re-run using standard 1U cable management bar:

Network leads patched in (2 for each ESXi server, 1 for the WHS 2011 server, 2 for IPMI and 1 running back to the house- hopefully to be replaced shortly by 2 fibre LAG):

And covered up:

Front of the rack (bit of a rubbish photo, but you can see none of the equipment has any front connections, also quite a lot of redundant gear):

L

Mac OS X Lion on ESXi 5

After countless failed attempts I’ve finally managed to get Mac OS X Lion running in ESXi 5.

I used Donk’s ESXi 5 Mac OS X Unlocker: http://www.insanelymac.com/forum/index.php?s=&showtopic=267296&view=findpost&p=1745191

Unfortunately when you try and boot from the Lion installation DVD the Virtual Machine hangs on the apple logo. Pressing F8 or configuring the VM to force entry into the BIOS on next boot allows you to select EFI Internal Shell

And boot verbosely by issuing the command: boot –v

We now see the boot is hanging at the PCI configuration stage:

Issuing the command: boot –v npci=0x2000

Allows us to get past the PCI configuration step but now hangs looking for the installation media (still waiting for root device):

This appears to be because the IDE controller is unsupported. Attaching an external USB DVD ROM, enabling pass-through:

Then attaching the USB controller to the VM:

And voila- the Lion installation begins!

The next obstacle came when trying to select the target disk for the installation. The virtual disk wasn’t listed and attempting to partition/format the disk resulted in an error: Unable to write to the last block of the device

I found a few suggestions: https://discussions.apple.com/thread/3226425?start=0&tstart=0

Launching Terminal and issuing: diskutil list
Allowed me to identify the disk: /dev/disk0
And issue: diskutil eraseVolume jhfs+ "OSX" /dev/disk0
Which also failed but back in the disk utility I was now able to partition & format the disk ready for the installation!

Everything went smoothly from here. Installing VMWare tools v5.0.0: http://www.insanelymac.com/forum/index.php?showtopic=267339 went smoothly but after rebooting I couldn’t login to (the password seemed to be being rejected). I rebooted the VM in safe mode (hold the shift key whilst during boot) and was able to login.

Performance is pretty poor (I think this is due to the lack of graphics acceleration) so I went straight in and enabled remote management so I could now use Apple Remote Desktop to administer the server.

Good luck!

L

*EDIT* One important thing to note the VM cannot be powered on from vCenter (error: Failed to find a host for powering on the virtual machine. The following faults explain why the registered host is not compatible. The guest operating system ‘darwin10_64guest’ is not supported). Simply logging directly into the host allows you to power on the VM.

I felt this deserved a separate post as the issue could have driven me crazy/made me return the motherboard in error!!!

After building the first server for my new Virtual Lab i installed ESXi and started to deploy my first VM. Everything was going nicely until vSphere lost the connection to the server. I used the IPMI remote control to take a look at the console and apparently both network interfaces were disconnected? Maybe something went wrong somewhere…

Reboot… Same problem after 5minutes or so the network connections drop
Reinstall ESXi… Same problem…
Recrimp a couple of new network cables… Same problem…

Finally i found this article which quite correctly identifies Active State Power Management as the cause! Pop into the bios:

Advanced, Advanced Chipset Control and disable Active State Power Management

Voila! Everything’s now chugging along nicely (fingers crossed)

L

%d bloggers like this: