Registering a VM created by a slightly newer version of ESXi

Related to the previous post, I’ve downgraded one of my ESXi hosts to ESXi 7U1c in a fit of frustration and sleep deprivation.

Now comes the headache of re-configuring this host, along with registering VMs that were created by a newer version of ESXi 7. Specifically, 7U2.

Et eu, VMs?

Now I didn’t really use any special 7U2-specific configuration for my VMs, so let’s go ahead and cheat our way into mutating their VMX configs so we can register them.

  1. Remove invalid VM from inventory
  2. Enable SSH on ESXi Host
    Lots of guides online for this already – use your favourite method, via vCenter, via ESXi web UI, via DCUI, etc.
  3. Open a vi session to your VM config file
    Assuming you’ve got your VMs tucked away in some datastore located in /vmfs/volumes, go ahead and do a vi /vmfs/volumes/<DATASTORE_NAME>/<VM_NAME>/<VM_NAME>.vmx
  4. Edit the virtualHW.version
    In my particular case, I dropped it from 19 to 18.
    List of virtualHW versions: https://kb.vmware.com/s/article/1003746
  5. Register VM

VMware ESXi 7U2 – Host losing access to SD card

Since Fall of 2021 when I upgraded from ESXi 7U1c to 7U2a, I’ve noticed several times when my host would lose access to its internal SD card. My setup is pretty standard – Dell’s custom ESXi image installed on my R830’s internal dual SD module (IDSDM).

After around 16-20 days of uptime, the host would lock up – API would be unresponsive for ESXi (busted vmware-hostd and/or vmware-vpxa services?) meaning that any attempt to manage VMs via ESXi web UI would timeout and/or sit there indefinitely, along with any attempts via esxcli. Hosts managed by vCenter would timeout for any dispatched vCenter jobs, with similar UI symptoms as above.
Some of these include:

  • Unable to manage power state of VMs (operation pending infinitely)
  • Unable to vMotion/migrate VMs
  • Web/remote console does not work

Some host functionality is persisted, such as managing host services and enabling SSH, but it’s very limited what you can do at this point.

Fortunately, running workload (VMs, vApps, Pools) seem to be unaffected.

What’s going on?

This is a known issue as of VMware KB #2149257 where high frequency read operations to ESXi’s SD card (either single, or IDSDM) causes ‘SD card corruption’.

This is attributed to a new partition schema, where ESXi’s scratch partition located on the same media experiences high I/O. Hence why in the same KB, they suggest the usage of Ramdisk as a VMware Tools repository.

There are other theories as well which hint at a bug in the vmkusb driver, an issue VMware engineers are still looking into.

It’s important to note that this has only been observed for those deploying ESXi to an SD card, IDSDM, or crappy flash drives. Those that have deployed ESXi to a disk (HDD/SSD) have not experienced this issue.

You can follow along with the VMware community discussion here.

What can I do?

You’ve got a few options, all of which have varying success and suck in production.

Ordered from safest to outage:

  1. Restart vmware-hostd and vmware-vpxa services, which seem to re-establish connectivity to the ESXi filesystem on your SD card.
  2. Move scratch partition to disk or datastore, and use Ramdisk for vmware-tools (done automatically in ESXi 7U3).
  3. Downgrade to ESXi 7U1c (last known 7.0 version before these issues)
  4. Export an ESXi config bundle backup, re-install ESXi, and restore from ESXi config bundle.

Depending on whether you’re experiencing one of the two aforementioned issues, YMMV.

How I temporarily remediated

Until we see this resolved 100% in the future, I’ve outlined a remediation plan based on feedback by other awesome individuals in the VMware community which should address both faults.

1. Stop vmware-hostd and vmware-vpxa

/etc/init.d/hostd stop
/etc/init.d/vpxa stop

2. Wait 60 seconds, attempt unload of vmkusb, and wait another 60 seconds

sleep 60
vmkload_mod -u vmkusb
sleep 60

3. Start vmware-hostd and vmware-vpxa 

/etc/init.d/hostd start
/etc/init.d/vpxa start

After a moment you should find your host is once again responsive.

4. Enter maintenance mode

esxcli system maintenanceMode set --enable true

You can also check maintenanceMode state, useful after the last step.

esxcli system maintenanceMode get

5. Enable Ramdisk for VMware Tools repo

esxcfg-advcfg -A ToolsRamdisk --add-desc "Use VMware Tools repository from /tools ramdisk" --add-default "0" --add-type 'int' --add-min "0" --add-max "1"

6. Reboot host

reboot

7. Exit maintenance mode

esxcli system maintenanceMode set --enable false

The TOHacks 2021 Digest

2021 was an eccentric one, riddled with changes from the widespread indoctrinate of remote work and study programs to virtual everything. Meetings, workshops and hackathons are but a few things we have had to deliver 100% digitally.

In spite of all this, we were not deterred. Our TOHacks 2021 hackathon ended up being one of our most successful virtual hackathons to date with over 700 participants and 190+ projects. We then followed up with TOConnect 2021 which delivered dozens of workshop sessions to foster the leaders of tomorrow.

This would not have been possible without the support of inspiring individuals dedicated to hacking a better future, our sponsors for their financial support and the many TOHacks volunteers that worked tirelessly in the background.

Here is a peek at some of the lesser known but equally grand projects our teams have been cooking in the background, along with their aspirations for the new year.

Cross-posted from Medium:
https://medium.com/@TOHacks/the-tohacks-2021-digest-7f00e00948bf

Collaborative hacking with Visual Studio Live Share

When hacking in a team, there are always some problem that devolves into peer programming. In a physical Hackathon, this is but a chair roll away. When virtual, your options become more limited. Either one person shares their screen at 5 frames per second over Zoom, or git is involved in ways it definitely should not be.

Introduced as an extension in 2018, Microsoft Visual Studio Live Share allows individuals to share a collaborative, live integrated development environment (IDE) like a Google doc. It also facilitates sharing of terminal sessions, web servers, and has integrated voice calling. All of this is available at no extra charge, irrespective of Visual Studio license, and works across both Visual Studio and Visual Studio Code.

You will encounter many peer-programming scenarios in your Hackathon journey — especially when virtual. Let us explore some common ones and how Live Share can help your team work as effectively and efficiently as possible — after all, the clock’s ticking.

Cross-posted from Medium:
https://medium.com/@TOHacks/collaborative-hacking-with-visual-studio-live-share-b1a9dd743c8a

Making Viable Hackathon Projects. The Secret? Unix Philosophy.

Two weeks ago, I partook in a hackathon revolving around Fast Healthcare Interoperability Resources (FHIR) — a standard defining the format of electronic health records. This experience was an eye-opener, a true deep-dive below the surface to the rest of the iceberg that is healthcare data. Using FHIR, teams were challenged to address particular problems in any way they pleased — as long as it leveraged integration to FHIR systems.

Our team spent the majority of the time playing around with the data — it was not until the last few hours that we addressed the problem and built a cute little vitals baseline dashboard. It was something clean, simple, and illustrated only what we wanted to — patient vitals and vital correlations.

We did not win unfortunately. There were a plethora of other really neat FHIR projects, leveraging all sorts of tech from machine learning frameworks to AI-powered parsers.

Either way, we packaged up our project and archived it away for later to fiddle with.

It was during a git commit that I began to wonder when I’ll be seeing this code again.
How many projects reach the end-of-the-line?
How many projects never seek fruition?

Cross-posted from Medium:
https://medium.com/@TOHacks/making-viable-hackathon-projects-the-secret-unix-philosophy-bf599ef88f07

Mastering your Tech Internship by Professionally Integrating

Not long from now we will be seeing a surge of new interns enter the tech space for winter 2022, along with a surge of fresh hires starting their IT journeys.

If you have reached this far, congratulations! You have already convinced your employer of your unique talents that make you best suited for this role. Keep that ball rolling and show the rest of your team why you belong there as well!

Make a strong first impression, arrive early, and submit deliverables on time — all of which are generic advice heard at the beginning of employment. Let us skip the boring-alities and head straight to some of the meat and gravy that will make you highly successful, fast.

At the core of it, both leadership and your team members are looking for integration. How well do you fit in on a daily basis? Can you work together with the team on similar problems while bringing your own creativity to the table?
If you got up and left tomorrow, will the team even notice?
 Your worth on any team must be visible — highly specialized/siloed initiatives do not count.

For this piece, we will dive into what some of the technical pieces you can embark on to reach new levels of involvement that will build a strong foundation to help you integrate with your team, and the overall organization.

Cross-posted from Medium:
https://medium.com/@TOHacks/mastering-your-tech-internship-by-professionally-integrating-97ed49d18801

Code execution cannot proceed because wlanapi.dll was not found

Encountered this issue during the install process of Mozilla VPN’s Windows 10 client to a Windows Server 2019 virtual host.

During install, the Mozilla VPN MSI threw the following error:

Service 'Mozilla VPN (broker)' (MozillaVPNBroker) failed to start. Verify that you have sufficient privileges to start system services.

This looks like a post-installation step, since all necessary binaries are already located in C:\Program Files\Mozilla\Mozilla VPN

Running C:\Program Files\Mozilla\Mozilla VPN\MozillaVPN.exe resulted in the following error, which immediately hinted at the issue:

The code execution cannot proceed because wlanapi.dll was not found. Reinstalling the program may fix this problem.

I’m installing this client in a Windows Server 2019 VMware VM, which does not have a wireless NIC. It does not look like Wireless LAN Service was installed either.

After installing the service through Server Manager’s “Add Roles and Features Wizard” (note this will require a restart), I can successfully complete installation, and start the Mozilla VPN service.

To resolve, install Wireless LAN Service windows feature

TOConnect 2021 finale - what’s the word?

Four score and several nights later, TOConnect 2021 comes to an end. As for participants, their journey has just begun - whether that’s continuing education, entering the workforce, or starting a new project of their own.

Nonetheless, the TOHacks team is proud to have been a part of it by imparting knowledge and valuable skills through various mastermind workshops from data science and machine learning to demystified tech and tech diversity. We also held project management and personal branding workshops, fueling the entrepreneur within.

Cross-posted from Medium:
https://medium.com/@TOHacks/toconnect-2021-finale-whats-the-word-ca99a1743e4

Half-way checkpoint @ TOHacks 2021 Bootcamp

As week 2 of TOConnect 2021 comes to a close, so does the realization that we’re half-way through.

Seems like it was only yesterday when we were learning about data transformation with Data Pipeline from Microsoft, and roasting resumes with SET.Hacks().

Let’s have a peek at what else we’ve been up to 🙂

Cross-posted from Medium:
https://medium.com/@TOHacks/half-way-checkpoint-tohacks-2021-bootcamp-70ebf8bd1a09