Forcefully stop hanging Foreman tasks

Download PDF

After a system crash I recently had the problem that Foreman had several orphaned tasks that could not be stopped from the web interface:

Because I planned a Foreman upgrade this was pretty bad as it prevented the pre-upgrade check:

# foreman-rake katello:upgrade_check
...
Checking upgradeability...
Checking for running tasks...
[ERROR] - There are 8 active tasks.

To fix this it is necessary to kill the orphaned tasks with fire. To do so, you will need to retrieve the task ID from the PostgreSQL database:

# su - postgres 
$ psql foreman
foreman=# select id, label, started_at, state, result from foreman_tasks_tasks where state != 'stopped';
                  id                  |                     label                     |     started_at      |  state  | result
--------------------------------------+-----------------------------------------------+---------------------+---------+---------
 e140a579-a4a2-4a5f-9c32-97b6ae3700e8 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-01-11 12:38:12 | paused  | error
 e955c5a3-8367-4cea-b468-14f89416eaea | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-04-12 08:42:00 | paused  | error
 73ac6137-e4f3-4092-940f-1b0fda065b63 | Actions::Katello::Host::GenerateApplicability | 2017-04-30 11:36:18 | paused  | error
 45ecf092-30f5-4d4d-b8db-4bb920fe6bec | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-05-04 08:57:57 | paused  | pending
 db1711f0-80d8-4cae-8659-c81918c0db25 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-05-08 07:08:45 | paused  | error
 4ca57d8f-8b1c-4837-ae1a-580a775f51a8 | Actions::Katello::Host::GenerateApplicability | 2017-08-16 21:12:37 | paused  | error
 b96b45f7-4c75-4a66-9216-04bfd8de7b50 | Actions::Katello::Host::GenerateApplicability | 2017-08-17 02:31:56 | paused  | error
 5d3aff81-f169-47c9-adba-752d074a3400 | Actions::Katello::Host::GenerateApplicability | 2017-08-17 10:56:08 | paused  | error
 4faa5498-aeaa-462f-9c0c-f2ddfe2d403d | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-08-19 17:08:43 | paused  | pending
 56509b4c-16e1-4167-bcf1-20682cbd23cb | Actions::Katello::EventQueue::Monitor         | 2017-08-19 17:08:44 | paused  | pending
 404cf497-6cc2-4fbd-8016-cb04fa1423e7 | Actions::Candlepin::ListenOnCandlepinEvents   | 2017-02-21 10:23:01 | running | error
 8e1be320-bead-4c64-a477-b46604c2bcd7 | Actions::Katello::Host::Erratum::Install      | 2017-03-10 14:44:30 | paused  | error
(12 rows)

The first column (id) of the dump shows the task ID; the third column (started_at) is also important. All tasks that were started before the last restart of Foreman need to be stopped.

It is a good idea to copy the appropriate IDs – the next step is to abort these tasks using foreman-rake:

# foreman-rake console
irb(main)> ForemanTasks::Task.find("e140a579-a4a2-4a5f-9c32-97b6ae3700e8").destroy
irb(main)> ForemanTasks::Task.find("e955c5a3-8367-4cea-b468-14f89416eaea").destroy
...

Afterwards it is recommended to check and fix database integrity:

# foreman-rake katello:reimport
...
Importing Katello::ContentViewPuppetEnvironment
Importing Activation Key Subscriptions

On older systems the command might differ:

# foreman-rake katello:reindex

Afterwards I was able to start the upgrade:

# foreman-rake katello:upgrade_check
...
Checking upgradeability...
Checking for running tasks...
[PASS] - There are 0 active tasks.
 You may proceed with the upgrade.
Download PDF