Solving VMware: The operation is not allowed in the current state of the datastore

At the customer site we installed a NetApp AFF A220 in their VMware vSphere environment. A very nice all flash array with tons of features and performance. To make reliable snapshots of their virtual machines we installed the Virtual Storage Console and NetApp Data Broker virtual appliances. But after the installation and configuration a weird problem occured.

Upon creating a virtual machine on some of the datastores the task failed with the error "the operation is not allowed in the current state of the datastore". During investigating I noticed that the datastore in question was overprovisioned although almost 50% free space was still available. Apparently all the great NetApp storage efficiency features like deduplication, compression and compaction where to much for VMware to comprehend.

When I was searching for the solution I stumbled on a VMware KB article suggesting to disable the NetApp components from vCenter. Obviously disabling the NetApp Virtual Storage Console and SnapCenter plugins did not solve the matter. Also removing them completely wasn't the solution.

Upon futher investigation I found that the datastore obtained an extra flag called thin_prov_space_flag. Below are the instructions to check for yourself.

First enable SSH access to your vCenter Appliance if not already enabled:
Browse to https:\\<your vcenter ip or hostname>:5480, login with root and enable SSH

Next, make a SSH connection to your appliance using your favorite client (eg. Putty) and, again, login with root. Run the shell command to enable the bash-shell.

login as: root

VMware vCenter Server Appliance 6.5.0.21000

Type: vCenter Server with an embedded Platform Services Controller

Using keyboard-interactive authentication.
Password:
Last login: Tue Jan 21 xx:xx:xx 2020 from x.x.x.x
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@<vcenter server> [ ~ ]# 

Before your are stopping services you might want to check if this also is your problem.
Find out your database password: cat /etc/vmware-vpx/vcdb.properties and connect to the database: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc

root@<vcenter server> [ ~ ]# cat /etc/vmware-vpx/vcdb.properties
driver = org.postgresql.Driver
dbtype = PostgreSQL
url = jdbc:postgresql://localhost:5432/VCDB
username = vc
password = <your db password>
password.encrypted = false
root@<vcenter server>[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc
Password for user vc:
psql.bin (9.4.15 (VMware Postgres 9.4.15.0-7748409 release))
Type "help" for help.
VCDB=>

Run the command to see the thin_prov_space_flag for the datastores: select id,thin_prov_space_flag from vpx_datastore;
(Don't forget the semicolon at the end)

VCDB=> select id,thin_prov_space_flag from vpx_datastore;
  id  | thin_prov_space_flag
------+----------------------
  151 |                    0
  163 |                    0
  157 |                    0
  143 |                    0
  169 |                    0
 3159 |                    1
 3160 |                    1
   15 |                    0
 3161 |                    1
 3134 |                    0
(10 rows)

VCDB=>

The datastore with the "1" under the thin_prov_space_flag are the problem datastores. The "1" prevents the virtual machines from being created. Now enter /q to leave the Postgress prompt and return to the bash shell. After that stop the Vmware services (service-control --stop vmware-vpxd)

VCDB=> \q
root@<vcenter server> [ ~ ]# service-control --stop vmware-vpxd
Perform stop operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False

Successfully stopped service vpxd
root@<vcenter server>[ ~ ]#

Again enter the PSQL prompt. When connected change the 1 to 0 for tha datastores: update vpx_datastore set thin_prov_space_flag=0 where id=<datastore id>; (again don't forget the semicolon) and repeat if necessary. When finished use /q to leave the Postgress prompt and return to the bash shell. After that start the Vmware services (service-control --start vmware-vpxd)

root@<vcenter server>[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc
Password for user vc:
psql.bin (9.4.15 (VMware Postgres 9.4.15.0-7748409 release))
Type "help" for help.
VCDB=> update vpx_datastore set thin_prov_space_flag=0 where id=3159;
UPDATE 1
VCDB=> \q
root@<vcenter server> [ ~ ]# service-control --start vmware-vpxd
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
xxxx-xx-xxTxx:16:50.306Z   Service vpxd state STOPPED
Successfully started service vpxd
root@<vcenter server> [ ~ ]#

Optionally you can check if the flag was correctely set to zero before leaving the Postgress prompt. select id,thin_prov_space_flag from vpx_datastore;

VCDB=> select id,thin_prov_space_flag from vpx_datastore;
  id  | thin_prov_space_flag
------+----------------------
  151 |                    0
  163 |                    0
  157 |                    0
  143 |                    0
  169 |                    0
 3159 |                    0
 3160 |                    0
 3161 |                    0
   15 |                    0
 3134 |                    0
(10 rows)

VCDB=> \q
root@<vcenter server> [ ~ ]#

Now virtual machines can be created again. So get back to you vCenter server and start provisioning.