Category: VMware

VMworld 2018 EU – How many vExperts can you fit in a photobooth?

It’s been a while since I have posted, for a number of reasons really. The main excuse is that I haven’t been doing much with VMware for my job in the last few months and I don’t want to post anything that I didn’t feel relevant. I’ve been having time with family too, which is the most important thing.

I’m back at VMworld 2018 EU this year after entering a competition on Twitter and winning. Completely unexpected this year and a bit last minute, I love the power of the community although it was a bit of a mad rush to get everything locked in.

Last night I had some great fun at the Rubrik party, met many old friends and some new. A highlight of the evening was “how many of the London VMUG could we fit inside the photobooth”. It turns out not too many, but we managed around 5 or 6.

Disclaimer:- The quality of the pictures are down to poor light and a bad front facing camera. They by no means represent or infer the inebriation levels of participants.

Feeling fresh and ready for the keynote this morning, we headed down and I was lucky enough to sit with some good friends and in fact housemates for the week @GarethEdwards86 @AndyNash99 @Dark_KnightUK in the blogger area with great views for the keynote.

I’m not going into all detail of announcements and VMware progress because others will certainly cover it (I did make extensive notes I promise, but on a whim have gone a bit rogue with this post). However, some top things:

– VMware 20th Birthday!
– VMware green initiative. Saving 540 Million tonnes of CO2 emissions, this equals Italy, Spain, Switzerland and the UKs output!
– VMware is now a carbon neutral company. Smashing the targets of 2020.
– “Possible belongs with you” using Tech as a for good.
– Daymn that NSX HTML 5 interface looks swanky!
– Heptio acquisition. Sharing a common mission of delivering “dial tone” kubernetes.

The things that resonated most with me today were Pat talking about Green initiatives and using Tech for good. The case study of Mercy Ships; a company that is trying to change a global surgery crisis, operating the worlds largest floating hospital that provides free care to countries in desperate need around Africa. Having remote infrastructure that is stable, in the form of VxRail means they can provide services without having to worry about managing tech and focus on providing healthcare.

Today has been a great day and in fact I’m confident it’s going to be a fantastic week. I’ve met lots of old friends and fellow @LonVMUG members. The spirit of community is strong and it continues to grow every VMworld that I attend.

A great example for me today was stepping up to assist @Ericnipro who came to the community area to ask for help for a French chap who was struggling with a VMware/Veeam issue. With the invaluable aid of @mwpreston, we helped the guy as best we could with a workaround script that we ended up stealing from Reddit. Then I chatted and caught up with Mike to which he gives me a jar of his homemade maple syrup!! Totally made my day, thank mate!

It hit me this week that I’ve actually made some impact in the community albeit small (although it doesn’t always feel that way) just by meeting people and spreading the word of the programme. A few people have attributed me to them now being vExperts which makes me feel good about myself and the community.

Too many people to mention them all, you know who you are! Too many people I haven’t yet properly met, let’s do this!

vCSA Automated Backup Failure

Recently we have gone through the process of upgrading our Windows 6.0 vCenter Server with external SQL to vCSA 6.5. I must say now how good the entire process was from start to finish, VMware have really done themselves proud on that tool. Our environment isn’t huge but it is big enough that we thought we might see problems – but no!

Part of the migration work was to get backups up and runnign as they were with our Windows vCenter (if not slightly different/better). My understanding is that the supported method for backup is to use the VAMI interface and run a full “file dump” backup of the vCSA with which you can restore into any blank deployed vCSA and you are back in the game. We have a Rubrik for snapshotting but using the VMware method is of course supported and preferred.

The Issue

Upon using the VMware provided Bash Script we encountered the following error in the backup.log file that is produced:

“{“type”:”com.vmware.vapi.std.errors.unauthenticated”,”value”:{“messages”:[{“args”:[],”default_message”:”Unable to authenticate user”,”id”:””}]}}”

Further investigation showed further errors in the VAPI endpoint log

We could run a manual backup from the VAMI interface as the root user but just not using the bash script which is essentially using the VAMI API to curl a request to run a backup. The error above seems related to “” and being unable to validate the signing chain signature. Without further help there was no way I was going in to modify or look at that script on my own on a now Production vCSA.

I also created a seperate master user in the @vsphere.local domain to test running the backups but still had no luck.

I ran the script manually and the problem occured at the start of the POST to the appliances rest API.

The Fix

After speaking with several smart people in the vExpert slack channel, I raised a case with VMware support. I eventually received a response which told me to edit the following file:

There is a value that needed changing from:

To the following:

Be careful with the amendment, there is space indentation on the code and there must be exactly 8 spaces in from the new line

Then a simple stop and start of the applmgmt service to apply the fix:

Now the script runs perfectly daily to our backup respository. I believe this might become defunct in vSphere 6.7 as I think there is now a GUI way of scheduling backups!

vCSA 6.5 High Availability Configuration Error

Recently I have been experimenting with configuring the built-in vCSA 6.5 HA functionality. Upon reading the documentation found here. I set about the task of configuring a basic HA deployment.

The error I saw upon completing the wizard was:

“A general system error occured: Failed to run pre-setup”.

Unfortunately, there wasn’t much to go on in the vCenter logs via the web GUI so it was time to SSH into the vCSA and go digging around for some logs with a little more information. After a brief meander, I found the following log

The interesting contents of the log were spat out as follows:

Looking at the log, it seemed that insufficient privileges were given to the user trying to create vcha user (root!). I then remembered the recent issues that VMware have had with Photon and root passwords expiring after 365 days. I logged into the VAMI for the vCSA and tried to reset the password but I was given an error.

The fix, in this case, was to simply reset the root password of the user via the bash shell.

At this point I was able to login with the new password and then login to the VAMI and set the root password to never expire. You can also do it via the command line using the “chage” command on the root user.

After restarting the deployment the pre-checks ran successfully and the configuration continued!

Hopefully this might help someone who is trying to do something similar!

VMworld USA 2017 – Wednesday Breakdown

Day three at VMworld was a bit of a slow start for me, the Rubrik party was a late one and there was no keynote so I decided to rest up a little try and save my energy.

Hanging out in the community areas, which is the best part of the event, was high on the agenda. Early on in the day we swung by to see our favourite Cloud Cred lady Noell Grier . I gave her a bit of a hand doing some “booth babe” duty whilst Rob Bishop collected his Go Pro 5 that he won for completing a CloudCred challenge! Noell is an awesome lady and if you aren’t familiar with CloudCred then you should go to the site, sign up, follow her on twitter and get on it!

The main highlight for Wednesday for me was heading to the customer party. Thanks to #LonVMUGWolfPack shenanigans Gareth Edwards, Rob Bishop and I ended up wearing some very jazzy VMware code t-shirts. The concert was a blast and we had a great time, I really enjoyed Blink 182 despite not being allowed on the main floor. Here are some pics of the event:

(Credit to Gareth for some of these pictures, thanks dude!)

NSX Performance
Speaker: Samuel Kommu

Samuel starts by a show of hands and it seems that most of the audience are on dual 10Gbe for their ESXi host networking.

NSX Overview of overlays
There is not much difference between VXLAN encapsulation and original ethernet frames. Only the VXLAN header is extra.
With Geneve Frame format there is an additional options field (length) that specified how much more data can be packed into the header. This can be interesting as you can pack extra information within it. This then helps capture information on particular flows or packets.

Perfomrance tuning
Parameters that matter – MTU mismatch is a pain to try and figure out. There are two places you can set it: ESXi host and on the VM level. From a performance perspective the MTU on the host doesn’t matter unless you change it at the VM level too.

There is a large chance if you change the MTU you will change the performance on your systems. The advice is to change the MTU to recommended values. The reason for this is the amount of payload vs. headers goes down therefore you are getting more for your money.

The vDS MTU sets the host MTU as that is what the host is connected to. The underlying physical network needs the same MTU setting too. Fairly standard stuff but important to check and consider.
Optimizations on TCP/IP , sending a large payload without spending CPU cycles. This is TSO. The act of sending a 1MB file for example, doesn’t happen within the system but it happens on the pNIC when chopping it up.**

With ESXi 6.5 they have brought in LRO in a software LRO rather than having the physical hardware only having it. Now it is possible to leverage LRO without physical capability on NSX 6.5.
When RSS is enabled
– Network adapter has multiple queues to handle receive traffic
– 5 tuple based hash for optimal distribution to queues
– Kernel thread per receive queue helps.

Rx/Tx filters
– Use inner packet headers to queue traffic

Native Driver – vmklinux driver data gets translated to vmkernel data structure. The native driver decreases the translation between both. Meaning less CPU cycles used.

Each vNIC now has it’s own queue down to the pNIC, rather than sharing the same queue. This scales throughput accurately through to the pNIC. It is also now possible to have multiple queues per single vNIC to pNIC.

Compatibility guide

The HCL is an obvious place to start with checking versions to ensure they are all correct and in support. It is then possible to select the right settings so that you can receive the latest and correct drivers to download and install onto your hosts.

Traffic Processes
Traffic flows, E/W and N/S traffic. E/W means a logical switch communication within the same switch to other VM’s .This is usually the most amount of traffic, smaller amounts go out on N/S traffic flow and this also goes through NSX Edge.

Long flows:
– Designed to maximums on bandwith
– Logs
– Backups
Short flows:
– Databases, specifically in memory ones or cache layers.}

Small packets:
– Keep alive messages

Not all tools are able to test the latest optmizations. Make sure the tools are right for the job. Application level is often best but be aware.
Fast Path
When packets come in, a new flow, it has different actions depending on the header. This happens throughout the entire stack regardless of E/W N/S traffic.

When you see new flows that are similar type, fast path disregards the flows actions and fast tracks to the destination, with no hash table. This is for a cluster of packets that arrive together, the flow is hashed and then sent via fast path. This causes 75% less CPU cycles.

The session got quite deep at times and went way further than my limited NSX experience could take me. I’m also not a network admin by day either so if there are any mistakes in my notes I’ll correct them as I go.

VMworld USA 2017 – Tuesday Breakdown

Tuesday starts with excitement at VMworld 2017, the keynote beings…


Pat Gelsinger and Michael Dell take the stage for Day 2, the crowds are in anticipation of a great session.

Pat opens up by talking about support and GSS. In recent years it is the opinion of some that support has been an issue and that standards might not as be as high as previous years. Pat states that he is committed to being the best technology partner and hopefully this will drive change from the top down through VMware to improve this area. This is fantastic news!

Michael gives his thoughts on machine learning and quantum computing topics. He talks about the sheer number of devices available now and the IoT trend. Data is growing at an exponential rate and if we are able to overlay computer science and machine intelligence to this data we reach a tremendous age of humans and computers working together for some great possibilities. He believes we are the dawn of this era.

Pat comes out with a classic line summizing this topic of conversation:

“Today is the slowest day of technical evolution in the rest of your life”.

Pat and Dell have a great rapport, this much is clear from their discussions on stage. There is a small amount of banter between them which gets the crowd laughing. It’s moments like these that make the event more enjoyable to watch as it shows that they are just guys who are passionate about technology, not multi-billion dollar CEO’s.

VMware and Pivotal are also announced as platinum partners of the Cloud Native Computing Foundation.

Later, Ray O’Farrel joins the stage to a big applause, he is a peoples favourite. He jokes that VMworld this year is a bit like a rock concert. He asks the question, how do we build the products that they put forward for us, the customer. The main principals to this focus are:
1) Customers can take advantage of the most modern infrastructure available.
2) Pragmatic about how we consume the technologies. Robust, quality delivery.#
3) New consumption models and how things can be delivered “as a service”.
4) Developer friendliness, allow devs to leverage the infrastructure and applications via code.

Ray Demonstrates using a fictitious company “Elastric Sky Pizza”. They are using Project Leonardo to push forward the company with digital transformation. The question being how can this company use products from VMware to help this transformation?

The answer is VMware Cloud Infrastructure a Unified SDDC Platform. A dive into how this complete stack of technology delivers a complete digital transformation for the business. The most impressive thing to me in the demonstration that follows is the VMware App Defence product. It is also nice to see end-to-end use of the entire product stack. I highly recommend that people catch up on the this day two keynote!


The rest of my day after the keynote was spent catching up with fellow community members and wondering around the solutions Exchange. I met up with my buddy, and fellow @LonVMUG member Gareth Edwards and we had a flourish of creativity in order to try and win the Turbonomic competition. As luck would have it, I was judged as the most creative of the day and won an Nvidia Shield. Thank you so much!

Here are my winning entries and me collecting my prize.

In the evening, we went to the Pinball hall of fame for the vExpert party. It was great fun to chat with fellow dedicated community members! We had several drinks, some excellent food and played some old/new school pinball and video games!

The final end (or start) to the evening was attending the Rubrik Party at the Cosmopolitan Hotel/Casino. Gareth and I attended with our white VIP wristbands and went into a fully booked night club for the evening to watch Ice Cube and mingle!

A great end to the evening (at 4:30am)! Thanks to all who were out, especially a big shout out to my main Eric Lee who was on fire the entire evening. Such fun times and entirely the reason I love being a vExpert and party of the VMware community.