Deleting Stuck Executions

Deleting Stuck Executions

You are here:
< Back
  • Executions in pending state, even though there are sufficient available Celery workers to handle them
  • Executions stuck in cancelling or force_cancelling state

Stuck executions can prevent the deleting or modifying a deployment. Even worse, stuck system workflow can bring all workflows on a Cloudify Manager grinding to a halt.

To recover from that, the trick is to update the execution status manually to a “completed” status (such as terminated, cancelled or failed).

Method 1: Using PostgreSQL

  • Log in to PostgreSQL as follows:

    • Log into Cloudify Manager with SSH.
      • If you work in an HA environment, SSH into the active manager.
  • If you have sudo access, execute the following command:

    sudo -u postgres psql cloudify_db

  • If you don’t have sudo access, execute the following command:

    psql -U cloudify -W -d cloudify_db -h localhost

  • Query the database to ensure that you know what you’re doing:

    select status from executions where id='execution-id';

    (Replace execution-id with the bad execution ID. Keep the apostrophes.)

  • Update the execution:

    update executions set status='cancelled' where id='execution-id';

    That will update the execution to have a cancelled status.

Method 2: Using the REST API

You can send an HTTP PATCH request as follows:

4.x
 Expand source
curl --request PATCH -H "Content-Type: application/json" -H "Tenant: <tenant-name>" -u <username>:<password> -d '{"status": "cancelled"}' https://<manager-ip>/api/v3.1/executions/<execution-id>

 

3.4.x
 Expand source
curl --request PATCH -H "Content-Type: application/json" -d '{"status": "cancelled"} 'https://<manager-ip>/api/v2.1/executions/<execution-id>

 

  • Replace username and password with a username and password who can be authenticated against the REST service.
  • Replace manager-ip with the IP / hostname of the manager.
  • Replace https with http if your REST service is not SSL-protected (which should never really happen).
  • Replace execution-id with the execution’s ID.
  • Replace tenant-name with the name of the tenant to which the execution belongs.
Last Updated On August 27, 2018