Deleting Stuck Executions
- Executions in
pending
state, even though there are sufficient available Celery workers to handle them - Executions stuck in
cancelling
orforce_cancelling
state
Stuck executions can prevent the deleting or modifying a deployment. Even worse, stuck system workflow can bring all workflows on a Cloudify Manager grinding to a halt.
To recover from that, the trick is to update the execution status manually to a “completed” status (such as terminated
, cancelled
or failed
).
Method 1: Using PostgreSQL
- Log in to PostgreSQL as follows:
- Log into Cloudify Manager with SSH.
- If you work in an HA environment, SSH into the active manager.
- Log into Cloudify Manager with SSH.
If you have
sudo
access, execute the following command:sudo
-u postgres psql cloudify_db
If you don’t have
sudo
access, execute the following command:psql -U cloudify -W -d cloudify_db -h localhost
Query the database to ensure that you know what you’re doing:
select status from executions where id='execution-id';
(Replace
execution-id
with the bad execution ID. Keep the apostrophes.)Update the execution:
update executions set status='cancelled' where id='execution-id';
That will update the execution to have a
cancelled
status.
Method 2: Using the REST API
You can send an HTTP PATCH request as follows:
- Replace
username
andpassword
with a username and password who can be authenticated against the REST service. - Replace
manager-ip
with the IP / hostname of the manager. - Replace
https
withhttp
if your REST service is not SSL-protected (which should never really happen). - Replace
execution-id
with the execution’s ID. - Replace
tenant-name
with the name of the tenant to which the execution belongs.