Category: VMworld

VMworld USA – Day 3

Today has been an interesting day with the keynote which I blogged about earlier.
After that I gave myself some time to wonder around and get some work stuff done in terms of investigating some more vendors at the solutions exchange.

Then my next session of the day was 5 Functions of Software Defined availability with Duncan Epping and Frank Denneman. I’ve been reading their vSphere Clustering deep dive book recently and it is superb. This session was a great compliment to the my reading and also included some new features in vSphere 6.0 that aren’t in the book (yet!?).


This was a great session and I’ll post my notes for it below, as I did yesterday for Day 2. The guys are great speakers and really know their stuff. At the end I decided to go up and say hello, I spoke with Duncan at the signing for my Essential VSAN book yesterday and managed to get him to sign my clustering deep dive book, so I went and spoke to Frank and had a chat with him. He was a top guy and was happy to sign the clustering deep dive book and eluded to a new copy possibly arriving the future!


I then decided to go and watch Chad Sakac speak at the EMC Vendor booth, not for the purpose of winning any swag but because he is a well known charismatic and enthusiastic speaker; the rumours were definitely true.


I then attended a great session with GS-Khalsa and Smruti Patel on Architecting Site Recovery Manager. My notes for that will follow at the bottom of this post.


I then decided to head back to my hotel to get ready for the vExpert/VCDX party at the Julie Morgan Ballroom. This was a fantastic opportunity to meet guys in the community and also some very smart VCDX’s. It was a brilliant experience talking to guys who I follow on Twitter. Pat Gelsinger turned up and gave a great speech about community. I felt that it was an excellent touch, having someone of his importance, turning up to meet the people at the core of technical/evangelism outside of VMware. He even took a selfie with us all which I thought was hillarous and really showed that he is a down to earth guy.


I also met Eric Neilson from the vCommunity Podcast, he’s a very friendly and funny guy. He even offered to take me on a tour of the VMware offices in Palo Alto this coming Saturday. I’ll post about that if we do get the chance!

I then went home very tired but had an absolute blast!! Tomorrow is the VMworld big party at AT&T park, which I look forward to covering!

5 Functions of Software Defined availability – Duncan Epping and Frank Denneman

There are many parts to a modern DC: VM , Server, Mgmt., Storage, DC, Network but the reality that business owners care about is the Application!

vSphere HA
Most used feature, importantly the restart of VMs or applications.
Heartbeats are important to determine what has happened to hosts.
Admission control allows you to reserve resources. It ensures VM’s will restart after a failed host.
Host failures to tolerate, % based capacity or designated failover host.

In version 6, the VM Component protection (VMCP) comes in to assist with APD/PDL scenarios.

Enable host hardware monitoring. Then specify in the VMM section what to do in the event of PDL or APD.

Recommendations in 6.0 for maintenance:
– Disable host monitoring
– Make sure to have a redundant mgmt. network.
o Fewest hops to isolation address
o MTU size end to end the same
o Enable portfast on your switch
o Route based on Originating Port ID (active/standby)
o Failback set to No.
o Pingable gateway address
– Use admission control

vSphere DRS
– Load balancing and initial placement.
– Dependant on vCenter.
– Brokers resources between producers and consumers.
– Goal is to provide resources of VM demands.
– Resource control allow for resource allocation based on business drivers.
– Provides cluster management (Maintenance mode and continuity rules anti/affinity)

When using resource pools, try and assign resource pools reservations and then VM shares within the pool.

Storage IO Control
Quick fix method for detecting issues short term.
– Controls congestion in shared datastore
– Focussed on solving short term problem
– Enabled at a datastore level
– Detects congestion by monitoring avg. IO latency for the datastore
– Latency above threshold indicated congestion
– SOIC throttles once congestion is detected.
o Controls IO issued per host
– Based on VM Shares, reservations and limits.
– Throttles adjusted dynamically based on workload
o Idleness
o Bursty behaviour

Storage DRS
More of a long term fix for a cluster of datastores.
– Controls congestion on a datastore cluster
– Detects congestion
o SOIC monitors average datastore latency
– Storage DRS migrates once congestion is detected
o Capacity threshold per datastore
o I/O metric threshold per datastore
– Affinity Rules
o Default affinity for VMFK (VM on 1 datastore)

Storage DRS is now aware of storage capabilities through VASA 2.0
– Array based thin-provision
– Array based dedupe
– Array based auto tiering
– Array based snapshot
– Storage DRS integration with SRM

It is possible to set IOPS reservations on VM’s through API only.

It is a technology that has progressed a lot since its birth in 2003. From SPDS technology to long distance AKA cross-cloud vMotion announced in 2015.
– Supports GEO Distances (150ms)
o No WAN acceleration needed
– Maintain standard vMotion guarantees
– Various optimizations
o Batched RPCs
o BDP socket buffer sizing
o Congestion window/slow start handoff
o Disk lock handoff changes
– Disk handoff changes
o Restricted lock handoff
o Minimise disk IO

What happens if the switchover process (stun/un-stunned) is too long?
Consider using the advanced setting:
VMX Option = extension.convertonnew = “FALSE”

vMotion anywhere, across vCenter Server Boundaries
– vMotion across hosts without shared storage
– Easily move VMs across vDS, vS and folders and data centers.
o Simplifies vCenter Migration and consolidation
o Aligns vMotion capabilities with larger DC environments

Network IO Control
– QoS on the vDS layer.
– Allows you to partition physical network bandwidth.
– Applies to vNIC and vDS port group

It uses resource pools which enables shares, reservations and limits to ensure availability of resources.
In v6 NetOIC v3 allows configuration of bandwidth requirements for individual VM’s.
DRS is aware of NOIC and on initial placement is aware of network resources and is able to place based on this information.

Tip: Use NOIC on VM level for reservations, I.E Tier 1 applications only.

Architecting Site Recovery Manager to meet your recovery goals – GS Khalsa & Smruti Patel

Protection Group considerations

Protection groups can have a many to many recovery plan relationship.
1) vSphere replication based protection groups. It’s simpler than array based replication and not tied to underlying storage technology – therefore it doesn’t need to be identical like Array Based.
2) Array based protection groups have consistency groups which used Array based replication (E.G – VMAX to VMAX). VM’s map directly to the data store specifically.
3) In SRM 6.1 there is a new Storage Based levering storage profiles.

– High level of automation compared to traditional protection groups.
– Policy based approach reduced OPEX
– Similar integration of VM provisioning, integration and decommissioning.

You select the VM to a policy and the policy defines the storage and the protection with SRM automatically.

– More PGs = more granular testing/failover
o DR testing is easier
o Failover only what is needed
o Added complexity

– Less PG’s = less complex and lower RTO
o Fewer LUNS, PG’s and recovery plans
o Less flexibility


Active/Passive Failover – Dedicated resources for recovery
Active-Active failover – Run low-priority apps on recovery infrastructure
Bi-Directional Failover – Production applications at both sites. Each site acts as the recovery site for the other.
Multi-Site – Many to one failover. Useful for remote office/branch office.

New features
Stretched storage & Orchestrated vMotion
– The best ofboth stretched storage and SRM
o Support stretched solutions with SRM
o Orchestrate cross-VC vMoton
– Unified plan for Disaster avoidane, disaster recovery and mobility
– Zero downtime migrations for planned maintenance and Disaster Avoidance.
– Ability to non-distruptivley test Recveroy plans
– Enhanced reliability with active-active datacenters and dual vCetners
– Lower RTO in event of unplanned failsures.

SRM is a paired topology. It always needs a paired server back to a central site. Maximum of 10 SRM.
SRM has a rule for each VM can only be replicated and protected once. In a triangle replication, you can have A to B, B to C, C to A.

Multi-site guidelines
– Keep it simple
– Each VM can only be protected once
– Each VM only replicated once
– Utilize enhanced Linked mode.

Impacts to RTO
It is one of the most metrics when designing a recovery plan.
How long does it take to decide to failover?
Disaster strikes – how long do you leave it before invoking failover?

IP Customization
– Workflow without IP customization
o Power on VM wait for heartbeats
– Workflow with IP
o Power on VM with network disconnected
o Customize IP utilizing VM tools
o Power off VM
o Power on VM and wait for VM tools

Considerations for lower RTO
– Fewer Larger NFS datastores /LUNs are better.
– Fewer PG’s
– Don’t replicate VM swap files
– Fewer recovery plans

VM Configuration
– Install Tools on VMs
– Suspend VMs on recovery
– PowerOff VMs

Recovery Site
– vCenter Sizing – it works harder than you think
– Number of hosts (more is better)
– Enable DRS for cluster load balancing on recovery operation.
– Different recovery plans target different clusters

– Be clear with the business
o What is RPO
o Cost of downtime
o Application priorities
o Units of failover
o Externalities
– Do you have executive buy in?

– Do you have documented SLA’s
– Do your SLA’s clearly communicate the RPO, RTO and availability of service tiers.
– Are your SLA documents readily available to everyone in the company?
– Use service tiers
– Minimal requirements/decisions

vSphere Infrastructure Navigator – helps you figure out how your VM’s are interconnected which is useful.

Test Network:
– Use VLAN or isolated network to create a test network environment
– Different port groups can be specified for SRM test against actual failover.
– Work with network team to try and replicate some form of the production network

Test the DR Plan Frequently

VMworld USA – Day 3 – Keynote

I’ve decided to take a different approach today for the keynote. Yesterday I took in the awe of the big hall but this time I’ve decided to head to the hang space and set up shop early to get a seat in front of the big screen.


Today’s main speakers are Sanjay Poonen, Martin Casado and Pat Gelsinger

Sanjay Poonen

Sanjay is the first to enter the stage this morning. He foresees that mobile devices are going to be the future and the remote control to most things in our lives, this is going to be the focus from VMware – any device any where. He goes on to mention that VMware are the market share leader in the EUC space and growing fast. Mobile is the new desktop.

He announces a guest from Microsoft, Jim Alkove joins him on stage to discuss customer benefits. Jim talks of the Windows 10 program and looking at bringing enterprise mobility to devices/users. The importance of security are mentioned, features such as device guard and credential guard. Identity protection is vital moving forward for users – this is important and come back to Sanjays point about devices being the future.


The spirit of the new world in Enterprise IT is highlighted that big companies are working together to optimize products for existing customers get the benefits from their infrastructure.

Sanjay then introduces Noah Wasmer. He demonstrates a brand new consumer device connecting to the cloud via the simplicity and management of AirWatch. It is now possible to combine AirWatch and AppVolumes to deploy software in a physical context to the end user device. The big announcement that these two products are combined as a tech preview known as Project A-Squared. Exciting stuff.

The problems of legacy applications is addressed by being able to harness the business mobility provided by this solution and enable these apps to end users on any device, any where.

Martin Casado

Martin discusses challenges faced in IT. The first challenge is provisioning and configuration of infrastructure, especially in a shared environment. The second challenge is troubleshooting, without having end to end visibility of all components it can be difficult to understand where the problem even starts. The final challenge is security, once things are released they do not change as it causes risk. The release of NSX 6.2 has a lots of new features, moving forward the era of network virtualization and addressing many of the above problems.

Pat Gelsinger

Pat enters the stage and explains how he is involved at the path forward VMware. Without doing that, he understands that it is important to go back in time and see how things have changed in order to learn for the future.


He shows a slide for how mobile-cloud is connected in 2015 and future predicted statistics for Q3 2019.

Devices per person is on the increase with future predictions going up to a predicted 5.9!

It is estimated that 5 billion people will be connected in 10 years. There will be 80% of people connected in 2025, which means that we truly will be globally connected. The impact of this is going to be substantial to business, education, healthcare, etc. It will be interesting to observe how technology will progress all of these industries.

Pat goes on to mention how the Unified Hybrid cloud will be fundamental in providing a platform on which all industry can harness the power of the cloud. He also states how Edward Snowden has made a massive impact in the future of security in the cloud and how countries have concerns about data sovereignty, but how this can be addressed by Unified Hybrid cloud.

Some of my favourites quotes from his talk:

“A security renaissance has begun, seize the day”
“Taking risk, is the lowest risk”

It is clear to me, as Pat nears the end of his time on stage, why VMware saved the best until last. He has had so much experience in the world of IT and is a: very articulate, insightful, intelligent and passionate speaker. I also get an impression that he is a very down to earth guy and a geek at heart; I really hope I get a chance to meet him at the vExpert party tonight!

VMworld USA – Day 2

Today has been another busy day, it was a tiring one yesterday and I was definitely down for more of the same today. No rest for the wicked…

I had a bad start, missing the session I wanted to go to for AppVolumes with Andrew Johnson. This was my fault as I hadn’t given myself enough time before the session to work out that it was actually located in the Marriott hotel nearby. It’s a huge hotel with many floors and the signs at the lobby didn’t help me find where I needed to go. It wasn’t the end of the world however, as I found a session close by for vSphere Storage. I’ll post below some of the key points that I took away from this session.

The rest of my day has been quite exciting. I managed to meet a few vHero’s of mine today. I had my Essential VSAN book signed by the legends that are Cormac Hogan and Duncan Epping. Really nice guys, I wish I stood and talked to them more but I didn’t want to take too much of their time and was a little star-struck, embarrassingly. Only after walking away I realised I could have had some conversations about VSAN, I even have some quesions for them!!! Oh well, I might see if I can catch them later. Here is a picture of us at the book signing:


Moving on to meeting more excellent guys, I was walking out of the lunch hall and looked to my right and saw Chris Wahl! As I was on a roll I thought I’d say hello and thank him for his contributions to the community. He was a thoroughly nice chap, I’m looking forward to perhaps bumping into him again.

In the spirit of VMworld, as I was writing this very blog post in the VMworld Bloggers section, Mr. Joep Piscaer (@jpiscaer) sits down next to me. As expected, he’s a top guy and we had a great chat about Nutanix, VCDX and the nuances of Hyper-VCDX (ask him!). I have thanked him many times in my Nutanix blog posts, it was great to meet him in person!

Finally to end the day before parties, I went to the vCSA Best Practices presented by William Lam. I’ve blogged a bit about the vCS to vCSA in my home lab and am interested in converting other environments. This session did not disappoint and I came away with some extra info and also the tech preview of the new migration appliance #migrate2vcsa.

I took notes on both the sessions I attended today and they can be found below.

vSphere Storage: Meet the Experts (VMFS, NFS, PSA, iSCSI) – Christos Karamanolis

ESXi 6 has improvements in the storage stack.
– Multiple completion worlds in PSA
– Deferred pool execution in PSA
(It is off by default expert mode advanced config can be enabled.
– Increases parallelism in parts of the storage stacks (Issuing compete I.O commands)
– Most beneficial interfaces with large cost per issuing IO (AHCI)

Performance tweaks for PCIe SSD
– Use multiple PVSCSI adapters – the more controllers you have the more aggregated IO you can have down to the hypervisor.
– Increases PVSCSI queue depth to 255
– Increase maximum disk requests to 255
– Use higher end configs with more than 6 cores per socket.
– Enable SCSI Deferred pool via ESXCLI – off by default as above.

iSCSI Improvements

– IPv6 support
– Software, Independent and dependant iSCSI and also booting from IPv6 target
– Independent HBA support DHCP through UI and ESXCLI
– Software iSCSI now has static route support. Routing with port binding is supported.

NFS improvements

Support starting vSphere 6

– Mount an NFS volume as NFS v4 or v4.1 NOT BOTH
– Supports multi-pathing for servers supporting session trunking
– Supports Kerberos authentication protocol to secure communication
– Supports non root users to access files when used with Kerberos
– Supports iPV6 with AUTH_SYS, not with Kerberos yet
– Upgrade path – using svMotion or unmount from NFSv3 and mount as v4.1 volume.

VMFS Updates

– Shared disk clustered file system optimized for MVs
– Enable advanced capability’s like snapshots , thin provisioning, DRS/SDRS.

Note: VMFS-3 is DEPRECATED starting vSphere 6.0 can no longer create them and upgrade is strongly recommended.


512e/4kn disk support

– Metadata aligned and sized for 4k sector disks
– Avoid RW on 512e disks for metadata I/O

Fast creation of large thick files
– Swap files and thick files
– Allow faster boot up of monster VMs
– Minimize I/O when accessing large thick files

Support for volume larger than 64Tb
Support for file larger than 64Tb

VMware vCenter Server Appliance – Best Practices & Tips/Tricks by William Lam


Hosts: 1000
Power on VM’s : 10,000
Hosts per Cluster: 64
VMs per cluster: 8000
Linked mode: Yes

These configuration maximums are true for both vCS and vCSA.

– In vCener 6.0 U1 the FULL VUM capabilities are in the vSphere web client.

– There 3 Deployment types: Embedded (PSC and VC) or standalone for either PSC or VC

When considering which deployment type to choose, it is worth knowing that the external PSC enables enhanced linked mode. This is a key consideration. If not then maybe keep things simple with embedded.

You can have a windows external PSC and a vCSA PSC replicate between each other seamlessly.

A common question is “Can I change from Embedded PSC to external PSC? The answer is in Update 1 , it will be possible to migrate from embedded to External PSC! The steps, simply put, are:

1) Deploy an external PSC
2) Replicate from embedded vCSA.
3) Point the VC component on embedded to the external PSC.
4) Use a reconfigure option using cmsso-util via CLI on the vCSA.

Note: This cannot be used to merge between two different SSO Domains.

In update 1 you can now deploy a vCSA to a vCenter or ESXi host, supporting brownfield and greenfield deployment types.

When deploying it is important that proper DNS is configured and reachable from Client desktop and vCSA and all IPs are resolvable for forward and reverse lookups.

Ensure proper NTP time synchronization. Check time on ESXi host, PSC and Client desktop are all sync!

If you are using a VDS with your vCSA, the portgruop it uses must be ephemeral when using ESXi as a deployment target.

vCSA Upgrade:

I posted about full releases and upgrading from 5.5 to 6.0 requiring a migration. BLOGGED.

For smaller Update releases (U1, U2, etc) there is now in-place upgrades available, just mount the ISO to the vCSA and it will upgrade with a brief service restart, causing minimal downtime.

vCSA Configuration.:

Services can now be managed from the Web UI with useful Options to stop/start/restart services on the vCSA.

The VAMI (Admin Interface) has retuned in 6.0 Update 1 with a brand new HTML web interface which still runs on port 5480.

There is a brand new PSC interface UI in U1 that also uses HTML5.

Everything that can be done from the VAMI interface can be performed from the appliance shell (applancesh).

The appliancesh is now automation friendly by being able to accept multiple commands via CLI.


It is possible to change the deployment size with extra memory and cpu. Dynamic memory resize support with a reboot.

Easy to resize disk capacity now by increasing the VMFK size. Step 2 run vpxd_servicecfg command line to expand the disk.


All configuration parameters about how the appliance is deployed in /etc/vmware/install-defaults

VIMTOP can be used on the vCSA to view services, similar to TOP.

Support bundles available via SSH, or open a browser to be able to download: https://VC-IP/appliance/support-bundle


There is a fling for vCS to vCSA converter which I also posted about.

Supports 5.5 to 5.5 at present and supports MS SQL Server 2008 R2.

There is also a Tech Preview of an improved appliance to perform the conversion which will hopefully be coming soon!

VMworld USA – General Session Day 2

As the crowds pour into Moscone North, it becomes evident that this event has a scale that I had not quite managed to comprehend. This is easily double the size and experience I had of VMworld EU, if not more. I arrived around 9:30am and the seats were going fast. I wasn’t too sure where to sit so I made my way to the press/blogger section – of course, everything was already full. Luckily I managed to squeeze my way into the front and get a pretty good seat by the stage (somehow forcing my way into the middle of a row of fully occupied people).


There is a buzz in the air as people are anticipating the start and announcements to come. The noise of people talking is loud and constant and is only silenced by the PA announcement that the event is about to start imminently. I’m fairly confident I can hear the wireless access points dying a death above me too, they definitely didn’t sign up for this barrage! Offline blog post writing for me it seems!




The session starts with a on-screen video presentation for the cloud academy promoting the unified hybrid cloud for applications. Then the VMware President and CEO Carl Eschehbach enters the stage accompanied with some cloud native apps.


He welcomes the audience and informs us that this is the biggest VMworld to date, with 23,000 attendees and over 50,000 people watching the live event unfold. There is a special mention to 21 guests who have been to all 12 VMworlds to date, quite an achievement!

Carl emphasises the importance of the destination give back scheme which is going to unlock the power of the VMware foundation to donate to some worth charities. I’ve taken my picture earlier in order to promote the scheme, I urge anyone reading this now who is at the conference to do the same! It’s quick and easy and will ensure that some worthy causes are given a helping hand.

The “Ready for Any” statement of this years conference is in order to enable us in IT to: RUN , BUILD, DELIVER and SECURE any challenges that might come in the future.

VMware are focussing on the one-cloud option, partnering with IT administrators to give seamless integration between public, private and managed service cloud; enabling the unified hybrid cloud which is seamless to user for on-premise and off-premise IT. Support for traditional application development and cloud-native applications is paramount to future development. These apps are useless unless users can gain access to the content from any device, anywhere.

Bill Fathers comes on stage after Carl and discusses application scaling and preventing latency from killing revenues with network bottlenecks. The emphasis here is that VMware is the best option for customers to protect their business “Crown Jewels”.

Yanbing Li announces the VMware EVO:SDDC Manager, which combines simplicity agility and availability at scale within the datacenter.


A great new demo of vMotion from Yanbing showing the migration of an LDAP services VM from on-prem to vCloud Air, it’s very impressive. The newly announced feature of cross-cloud vMotion is well received by applause from the crowd and makes history.

This was definitely my favourite part of the session; excellent demo’s, superb speakers with fantastic presentation.

Ray O’Farrel and Kit Kolbert come out with 15 minutes to spare to talk about cloud-native applications and containerization driving business agility forward. The demo of developers using docker containers shows challenges of partial visibility from an IT Infrastructure perspective. A new product vCenter integrated containers enables visibility of containerized VM’s inside host VM’s. Abstracting the container VM’s from the guest host VM means that all management features any normal VM are now available to the container VM’s through the vSphere Web GUI.


The benefits of using Photon OS with Instant clone means that ultra low footprint, no memory overhead VM’s means that a “just enough” VM is created for container use.

Some great announcements and offerings from this general session, I’m really looking forward to tomorrow’s session!

VMworld USA – Day 1

Wow! It has been a hectic first day in the USA. I haven’t actually been here 24 hours yet and it has been a whirlwind!

I got into my hotel late last night and then the first thing I had to do was get up this morning ready for the VMworld 5k Fun Run. I was a little dubious about doing this as being a bit tired and jet lagged isn’t the best conditions to be out running but it was a great event, lots of fun and really worth doing! A totally surreal experience running on the Golden Gate Bridge. I’d like to thank the organizers for putting the event on and I’m glad that some of the proceeds went to charity. I got to meet some nice people during the preparations which is always a bonus.




After a busy morning I made sure I got back to my hotel, showered and got ready to explore the Moscone!

It seems apparent that during VMworld the conference and attendees pretty much occupy this area of San Francisco! There are people everywhere with lanyards and backpacks. The Moscone is separated into three main buildings and I took some time to familiarise myself with the locations and the different areas in each.

My first session of the day was the Expert led vRealize Automation 101 workshop (HOL-SDC-1621) which we ran through how to use the vRealize Code Stream to enable orchestration and automation into build, test and deploy scenarios. The lab is a long one so I only got to complete a few of the modules but it was a valuable insight into the power that DevOps can harness. This will be something I will login to at a later date and explore further!

I then had some time where I had no sessions booked and the Solutions Exchange had not opened. I decided to head up to the City floor of the Meteron building where @vmunderground were hosting some great talks. It was very quiet up there but there were many well known faces giving some great talks. This was my find of the day as I hadn’t planned on being there but had some great sessions which I tweeted about. Notably for me the backup and data protection one was very good and had an impromptu guest of @HowardMarks!

Finally, I took a walk over to the Solutions Exchange and was overwhelmed at how large it was. It has to be twice the size of the VMworld Europe exchange. Vendors as far as the eye can see!


I had some great chats with the guys @RubrikInc and will be following up with them later. Even if it is just to take a ride in the awesome Back to the Future Delorean.

I also got to meet the great @CodyHosterman who has helped me in the past with documentation on storage/SRM and also @AngeloLuciani at the Nutanix booth who is a great guy! Both people I really wanted to meet in person having worked with them online in some shape or form! A really nice thing to experience at VMworld.

I’m heading to meet some UKVMUG guys now, then I’m off to the VMUnderground party. My legs ache from the running this morning and all the walking I’ve done but I’m still having a blast! No rest for the wicked I guess. Looking forward to more sessions tomorrow!