I have over fifteen years of experience in systems and network engineering, specializing in Linux and Cisco based networking.
I have focused on the datacenter and cloud over the last 6 years. Managing webscale properties with as many as 1m active users and building for 10x.
I am extremely flexible with regards to working hours and location – however I am Wellington, NZ based and any move would need to be planned.
I have maintained my skills by being hands on in all roles I have worked in – this allows me to act in both a design (architect) and implementation (engineer) capacity. I also have experience managing people.
Nov 2014 - Current
This project is still under wraps. But we are working on a solution to make the world a simpler place. We are a company of 3 co-founders, 2 in Israel one in NZ, with R&D Based in NZ.
Sept 2014 – Oct 2015
In September 2014 I started a 6 month contract with 9spokes with the mandate of taking the infrastructure ownership inhouse and to provide visibility to the business of best current practice.
This role was a mix of design and hands on technical work (30/70).
- Infrastructure Migration – Due to a commercial partnership between 9spokes and IBM the decision was taken to migrate infrastructure from DigiWeb (Auckland and Sydney) and AWS (Sydney) to Softlayer (Melbourne and Sydney). This has involved the design and build of a layer 3 network inside Softlayer utilizing Brocade Vyatta software routers to provide similar functionality to AWS security groups. We also utilized Citrix XenServer as the hypervisor for compute.
- OS Migration – I have migrated around 100 VMs from AWS Linux (Amazon’s CentOS) to Ubuntu LTS, this was done to avoid any specific vendor lock in.
- Environmental Rethink – There were numerous infrastructure decisions made that would only work with AWS a large part of my role has been to unwind these so that any provider can be used.
- V2 Infrastructure – I was responsible for the build of the version 2 infrastructure for 9Spokes, this is based on a globally active/active deployment with presence in Melbourne/Sydney (APAC Pod) and Amsterdam/London (EU Pod). This infrastructure is provided using an OpenStack installation on SoftLayer bare metal.
- Saltstack Deployment – I have deployed Saltstack as the automation and configuration management system of choice. This includes full automation of the creation of complex services such as MySQL clusters with floating IPs. This work has made extensive use of salt-mine along side pillars and grains and has seen me contributing code back to the project.
- Nagios & Cacti Deployment – Nagios alerting via PagerDuty and Cacti statistics have been deployed across all infrastructure, the network is largely self configuring using salt.
- Syslog/ElasticSearch/LogStash/Kibana – Due to the nature of operating a web scale infrastructure we deployed a ELK setup. This is essential in any large infrastructure.
Mar 2013 – Aug 2014
In March 2013 I accepted the task of designing and building the new infrastructure platform and assembling the BAU infrastructure team at Koding. (Case study and Diagrams available on request.)
- Infrastructure Migration and US Datacentre Build– I was responsible for the initiation, design and migration of Koding’s infrastructure from a mixture of Softlayer dedicated and Amazon AWS to a private cloud running OpenStack. This project has reduced infrastructure costs by approximately $800k/month, and increased capacity 10 fold. We also built out burst capacity into Softlayer San Jose.
- OS Standardization – I standardized the company on Ubuntu LTS releases (around ~200 VMs) for the backend systems and built a large part of the App on Ubuntu Raring/LXC – we ported this work to Ubuntu Trusty/LXC.
- Deployment of Ceph – For the main Koding app I built a 6 node Ceph cluster. The VMs given to the users run as LXC isolated machines with their filesystems being AuFS mounted RBD volumes (Like docker, but not docker). The Ceph cluster had 252TB of raw storage on 3TB SAS disks – I also used FusionIO as the OSD journals.
- Chef to Saltstack migration – I was responsible for the migration from Chef to Salt, this included the evaluation of Ansible and Puppet against Saltstack.
- European Datacentre Launch – I was responsible for all design and deployment of the new Koding European datacentre.
- Global Network Build – To support our growing datacentre footprint we have invested heavily in a global L2/L3 network. This includes direct allocations from ARIN and direct peering at a number of IXPs. The network was built using Nexus (5000/7000) switching in the Datacentres, Catalyst switching in the WAN edge and Mikrotik routing (BGP/OSPF) over MPLS.
- Syslog/ElasticSearch/LogStash/Kibana – I first deployed ELK at Koding in late 2012, working directly with one of our software engineers we built a visualization and error handling system to make the large microservice based system essentially self healing/diagnosing.
- Day to day management of 3 staff and over 1,000,000 user VMs
Aug 2011 – Mar 2013
- Firewall refresh – I joined this project around 50% of the way through, the project was to migrate 8 Checkpoint R64 firewalls (4 clusters) (Nokia IP2450) to Juniper SRX firewalls and upgrade a clustered pair acting as VPN endpoints to R75 (Nokia IP280) along with a roadmap/plan to R76 and R77 for these VPN appliances. We had early access to R76 in the IBM Lab. I was responsible for significant rule validation and solution design, I also acted as the escalation point during cutovers.
- Westpac Datacentre Migration – I was responsible for the design and documentation of the migration activities for the WNZL security infrastructure. This migration exposed a number of issues with the design and implementation of their Juniper firewall platform which I worked closely with Juniper to resolve. I also was called in as the SME for a number of P1 network incidents related to the network migration stream. This involved remedial work on the Nexus based network (2000/5000/7000) and Adva DWDM.
- Tools server platform – I was responsible for all layer 2 and 3 design for the new tools platform delivered to WNZL. This included all firewall and security flow design and documentation. This project was to deliver a new suite of cloud based tools for the management of WNZL’s infrastructure.
- VMWare refresh – I was responsible for all layer 3 design for the new VMWare platform. I also assisted with resolution of multiple layer 2 issues observed during this project.
- Contact energy network design – I was the key SME engaged
Dec 2009 – Aug 2011
In December 2009 I ventured out on my own and started c10k Consulting to serve some clients I had gained over the years. I have also been engaged on many occasions to fill knowledge gaps within many organizations when they have the skills to keep a system running but simply need help and guidance during initial deployment and times of upgrade. I was responsible for the day to day management of many servers across many different public clouds, including AWS, Rackspace, Softlayer, SliceHost and FDC.
- www.missuniverse.com – I was an ad-hoc systems admin for MUO during their events when traffic can reach 180,000 concurrent sessions, during this time I was responsible for the design and deployment of a number of auto scaling cloud systems.
- www.hbdirect.com – I was responsible for all aspects of the HBDirect web platform, I focused on the systems side but it also fell to me to engage outsourced developers for major code changes.
- www.oscars.org – I was brought in to help them with an upgrade to their DAM (Digital Assets Management) system Artesia, this system runs on Solaris based Oracle DB (RAC) servers and JBoss application servers.
- Reliam – I was assisting Relaim with ad-hoc systems administration services, this has included deployments of a number of large clients and the support for them.
I have been involved with many other clients during this time and been involved with many exceptionally interesting projects. Often I will be engaged for a few weeks and then sporadically after that usually for upgrade advice.
Sept 2009 – Dec 2009
Fishpond were recognized as the leading Internet based bookstore in New Zealand and Australia. I joined fishpond at a key time to get them ready for the Christmas rush.
- I was responsible for the planning, design and implementation of the new office network, being an online business internet connectivity is of paramount importance to fishpond, the final design involved redundancy at every level.
- I was responsible for upgrading the webservers from single server systems to horizontally scaling load balanced systems in Amazon’s AWS.
- I was responsible for upgrading the MySQL servers from version 4.x to 5.x – fishpond had 15+ MySQL servers running in a Maser/Slave setup.
- I deployed Puppet v0.25 to orchestrate the EC2 infrastructure.
Sept 2008 – Oct 2009
iSERVE were recognised as the largest hosting provider in New Zealand, I had overall responsibility for over 3000 servers, consisting of approximately 1200 Linux servers and 500 Windows servers. I was the senior technical resource at iSERVE for both networking and systems (storage and servers). This involved mentoring the level 1 and 2 staff, and being the go-to guy for all technical aspects of the iSERVE operations.
- I was responsible for the planning, design and implementation of the new iSERVE network. This consisted of replacing the firewalls (Linux IP Tables), Installing new routers (Cisco 7200 VXR), and installing Cisco subscriber control units (SCM’s). The core was built using a pair of Cisco 6500 VSS. The access layer consisted of 30 TOR switches.
- I planned and executed the migration from custom bash shell scripts to Puppet v0.25. This was deployed fleet wide.
- I was responsible for the design and implementation of the infrastructure and systems component of iSERVE’s new cloud server/virtual server offering. This platform was built around IBM HS22 Blades & IBM DS4300 SAN units the Blades run Citrix XenServer 5.5. This new platform was the first in New Zealand to offer true online provisioning of virtual servers in real-time.
- Day to day management of 2 staff and over 65,000 customers services.
Feb 2001 – Sept 2008
- I was responsible for the design and implementation of a 15 site WAN, this system was deployed using Cisco (800/1800/2800 Series) Routers running IPSec. The sites were a mixture of single user sites and branch offices. The connection technologies ranged from ADSL for the single user sites, to metro ethernet for the larger offices.
- I was responsible for the design and implementation of the companies SAN infrastructure, this was built using IBM FAStT700 SAN units, Brocade Silkworm switches and ADIC Tape libraries.
- I was responsible for the design and implementation of a high availability datacenter solution for the company.
- Day to day management of 2 staff