At the customer site we installed a NetApp AFF A220 in their VMware vSphere environment. A very nice all flash array with tons of features and performance. To make reliable snapshots of their virtual machines we installed the Virtual Storage Console and NetApp Data Broker virtual appliances. But after the installation and configuration a weird problem occured.

Upon creating a virtual machine on some of the datastores the task failed with the error "the operation is not allowed in the current state of the datastore". During investigating I noticed that the datastore in question was overprovisioned although almost 50% free space was still available. Apparently all the great NetApp storage efficiency features like deduplication, compression and compaction where to much for VMware to comprehend.

When I was searching for the solution I stumbled on a VMware KB article suggesting to disable the NetApp components from vCenter. Obviously disabling the NetApp Virtual Storage Console and SnapCenter plugins did not solve the matter. Also removing them completely wasn't the solution.

Upon futher investigation I found that the datastore obtained an extra flag called thin_prov_space_flag. Below are the instructions to check for yourself.

First enable SSH access to your vCenter Appliance if not already enabled:
Browse to https:\\<your vcenter ip or hostname>:5480, login with root and enable SSH

Next, make a SSH connection to your appliance using your favorite client (eg. Putty) and, again, login with root. Run the shell command to enable the bash-shell.

login as: root

VMware vCenter Server Appliance 6.5.0.21000

Type: vCenter Server with an embedded Platform Services Controller

Using keyboard-interactive authentication.
Password:
Last login: Tue Jan 21 xx:xx:xx 2020 from x.x.x.x
Connected to service

    * List APIs: "help api list"
    * List Plugins: "help pi list"
    * Launch BASH: "shell"

Command> shell
Shell access is granted to root
root@<vcenter server> [ ~ ]# 

Before your are stopping services you might want to check if this also is your problem.
Find out your database password: cat /etc/vmware-vpx/vcdb.properties and connect to the database: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc

root@<vcenter server> [ ~ ]# cat /etc/vmware-vpx/vcdb.properties
driver = org.postgresql.Driver
dbtype = PostgreSQL
url = jdbc:postgresql://localhost:5432/VCDB
username = vc
password = <your db password>
password.encrypted = false
root@<vcenter server>[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc
Password for user vc:
psql.bin (9.4.15 (VMware Postgres 9.4.15.0-7748409 release))
Type "help" for help.
VCDB=>

Run the command to see the thin_prov_space_flag for the datastores: select id,thin_prov_space_flag from vpx_datastore;
(Don't forget the semicolon at the end)

VCDB=> select id,thin_prov_space_flag from vpx_datastore;
  id  | thin_prov_space_flag
------+----------------------
  151 |                    0
  163 |                    0
  157 |                    0
  143 |                    0
  169 |                    0
 3159 |                    1
 3160 |                    1
   15 |                    0
 3161 |                    1
 3134 |                    0
(10 rows)

VCDB=>

The datastore with the "1" under the thin_prov_space_flag are the problem datastores. The "1" prevents the virtual machines from being created. Now enter /q to leave the Postgress prompt and return to the bash shell. After that stop the Vmware services (service-control --stop vmware-vpxd)

VCDB=> \q
root@<vcenter server> [ ~ ]# service-control --stop vmware-vpxd
Perform stop operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False

Successfully stopped service vpxd
root@<vcenter server>[ ~ ]#

Again enter the PSQL prompt. When connected change the 1 to 0 for tha datastores: update vpx_datastore set thin_prov_space_flag=0 where id=<datastore id>; (again don't forget the semicolon) and repeat if necessary. When finished use /q to leave the Postgress prompt and return to the bash shell. After that start the Vmware services (service-control --start vmware-vpxd)

root@<vcenter server>[ ~ ]# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc
Password for user vc:
psql.bin (9.4.15 (VMware Postgres 9.4.15.0-7748409 release))
Type "help" for help.
VCDB=> update vpx_datastore set thin_prov_space_flag=0 where id=3159;
UPDATE 1
VCDB=> \q
root@<vcenter server> [ ~ ]# service-control --start vmware-vpxd
Perform start operation. vmon_profile=None, svc_names=['vmware-vpxd'], include_coreossvcs=False, include_leafossvcs=False
xxxx-xx-xxTxx:16:50.306Z   Service vpxd state STOPPED
Successfully started service vpxd
root@<vcenter server> [ ~ ]#

Optionally you can check if the flag was correctely set to zero before leaving the Postgress prompt. select id,thin_prov_space_flag from vpx_datastore;

VCDB=> select id,thin_prov_space_flag from vpx_datastore;
  id  | thin_prov_space_flag
------+----------------------
  151 |                    0
  163 |                    0
  157 |                    0
  143 |                    0
  169 |                    0
 3159 |                    0
 3160 |                    0
 3161 |                    0
   15 |                    0
 3134 |                    0
(10 rows)

VCDB=> \q
root@<vcenter server> [ ~ ]#

Now virtual machines can be created again. So get back to you vCenter server and start provisioning.

4 Replies to “Solving VMware: The operation is not allowed in the current state of the datastore”

  • Edwin,

    Thnxs , had the problem that the customer could not storage vMotion a VM to a specific NFS datastore.
    Solved by your solution !!!

    Groeten
    Peter

  • Hi, thanks for this info. We also had the same issue and I showed this website to VMWare support. After gathering logs they came back a few days later with “yes we need to perform these steps”.

    One change to the commands that shows the thin_prov_space_flag was the following line:

    select name,storage_url,capacity,free_space,type from vpx_datastore where THIN_PROV_SPACE_FLAG=1;

    The output gives you the name of the datastore in question, just to be on the safe side before making changes.:

    name | storage_url | capacity | free_space | type
    —————–+—————————————+—————+—————+——
    NAC1_02_NFS_DS1 | ds:///vmfs/volumes/10b05310-38fbb534/ | 1649267441664 | 1128509902848 | NFS
    (1 row)

    So maybe something to consider adding to your steps for those that want that extra piece of mind 🙂

    Thanks again.

Leave a Reply