Search 1.9 billion lines of Odoo code on GitHub
queue_job
<h1 class="title">Job Queue</h1>
<p><a class="reference external" href="https://odoo-community.org/page/development-status"><img alt="Mature" src="https://img.shields.io/badge/maturity-Mature-brightgreen.png" /></a> <a class="reference external" href="http://www.gnu.org/licenses/lgpl-3.0-standalone.html"><img alt="License: LGPL-3" src="https://img.shields.io/badge/licence-LGPL--3-blue.png" /></a> <a class="reference external" href="https://github.com/OCA/queue/tree/14.0/queue_job"><img alt="OCA/queue" src="https://img.shields.io/badge/github-OCA%2Fqueue-lightgray.png?logo=github" /></a> <a class="reference external" href="https://translation.odoo-community.org/projects/queue-14-0/queue-14-0-queue_job"><img alt="Translate me on Weblate" src="https://img.shields.io/badge/weblate-Translate%20me-F47D42.png" /></a> <a class="reference external" href="https://runbot.odoo-community.org/runbot/230/14.0"><img alt="Try me on Runbot" src="https://img.shields.io/badge/runbot-Try%20me-875A7B.png" /></a></p>
<p>This addon adds an integrated Job Queue to Odoo.</p>
<p>It allows to postpone method calls executed asynchronously.</p>
<p>Jobs are executed in the background by a <code>Jobrunner</code>, in their own transaction.</p>
<p>Example:</p>
<pre>
<code lang="python">from odoo import models, fields, api
class MyModel(models.Model):
_name = 'my.model'
def my_method(self, a, k=None):
_logger.info('executed with a: %s and k: %s', a, k)
class MyOtherModel(models.Model):
_name = 'my.other.model'
def button_do_stuff(self):
self.env['my.model'].with_delay().my_method('a', k=2)</code>
</pre>
<p>In the snippet of code above, when we call <code>button_do_stuff</code>, a job <strong>capturing
the method and arguments</strong> will be postponed. It will be executed as soon as the
Jobrunner has a free bucket, which can be instantaneous if no other job is
running.</p>
<p>Features:</p>
<ul class="simple">
<li>Views for jobs, jobs are stored in PostgreSQL</li>
<li>Jobrunner: execute the jobs, highly efficient thanks to PostgreSQL's NOTIFY</li>
<li>Channels: give a capacity for the root channel and its sub-channels and
segregate jobs in them. Allow for instance to restrict heavy jobs to be
executed one at a time while little ones are executed 4 at a times.</li>
<li>Retries: Ability to retry jobs by raising a type of exception</li>
<li>Retry Pattern: the 3 first tries, retry after 10 seconds, the 5 next tries,
retry after 1 minutes, ...</li>
<li>Job properties: priorities, estimated time of arrival (ETA), custom
description, number of retries</li>
<li>Related Actions: link an action on the job view, such as open the record
concerned by the job</li>
</ul>
<p><strong>Table of contents</strong></p>
<div class="contents local topic" id="contents">
<ul class="simple">
<li><a class="reference internal" href="#installation" id="id3">Installation</a></li>
<li><a class="reference internal" href="#configuration" id="id4">Configuration</a></li>
<li><a class="reference internal" href="#usage" id="id5">Usage</a><ul>
<li><a class="reference internal" href="#developers" id="id6">Developers</a></li>
</ul>
</li>
<li><a class="reference internal" href="#known-issues-roadmap" id="id7">Known issues / Roadmap</a></li>
<li><a class="reference internal" href="#changelog" id="id8">Changelog</a></li>
<li><a class="reference internal" href="#bug-tracker" id="id9">Bug Tracker</a></li>
<li><a class="reference internal" href="#credits" id="id10">Credits</a><ul>
<li><a class="reference internal" href="#authors" id="id11">Authors</a></li>
<li><a class="reference internal" href="#contributors" id="id12">Contributors</a></li>
<li><a class="reference internal" href="#maintainers" id="id13">Maintainers</a></li>
</ul>
</li>
</ul>
</div>
<a name="installation"></a>
<h2><a class="toc-backref" href="#id3">Installation</a></h2>
<p>Be sure to have the <code>requests</code> library.</p>
<a name="configuration"></a>
<h2><a class="toc-backref" href="#id4">Configuration</a></h2>
<ul class="simple">
<li>Using environment variables and command line:<ul>
<li>Adjust environment variables (optional):<ul>
<li><code>ODOO_QUEUE_JOB_CHANNELS=root:4</code> or any other channels configuration.
The default is <code>root:1</code></li>
<li>if <code>xmlrpc_port</code> is not set: <code>ODOO_QUEUE_JOB_PORT=8069</code></li>
</ul>
</li>
<li>Start Odoo with <code>--load=web,queue_job</code>
and <code>--workers</code> greater than 1. <a class="footnote-reference" href="#id2" id="id1">[1]</a></li>
</ul>
</li>
<li>Using the Odoo configuration file:</li>
</ul>
<pre>
<code lang="ini">[options]
(...)
workers = 6
server_wide_modules = web,queue_job
(...)
[queue_job]
channels = root:2</code>
</pre>
<ul class="simple">
<li>Confirm the runner is starting correctly by checking the odoo log file:</li>
</ul>
<pre class="code">
<code class="code">...INFO...queue_job.jobrunner.runner: starting
...INFO...queue_job.jobrunner.runner: initializing database connections
...INFO...queue_job.jobrunner.runner: queue job runner ready for db <dbname>
...INFO...queue_job.jobrunner.runner: database connections ready</code>
</pre>
<ul class="simple">
<li>Create jobs (eg using <code>base_import_async</code>) and observe they
start immediately and in parallel.</li>
<li>Tip: to enable debug logging for the queue job, use
<code>--log-handler=odoo.addons.queue_job:DEBUG</code></li>
</ul>
<table class="docutils footnote" frame="void" id="id2" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>It works with the threaded Odoo server too, although this way
of running Odoo is obviously not for production purposes.</td></tr>
</tbody>
</table>
<a name="usage"></a>
<h2><a class="toc-backref" href="#id5">Usage</a></h2>
<p>To use this module, you need to:</p>
<ol class="arabic simple">
<li>Go to <code>Job Queue</code> menu</li>
</ol>
<a name="developers"></a>
<h3><a class="toc-backref" href="#id6">Developers</a></h3>
<p><strong>Configure default options for jobs</strong></p>
<p>In earlier versions, jobs could be configured using the <code>@job</code> decorator.
This is now obsolete, they can be configured using optional <code>queue.job.function</code>
and <code>queue.job.channel</code> XML records.</p>
<p>Example of channel:</p>
<pre>
<code lang="XML"><record id="channel_sale" model="queue.job.channel">
<field name="name">sale</field>
<field name="parent_id" ref="queue_job.channel_root" />
</record></code>
</pre>
<p>Example of job function:</p>
<pre>
<code lang="XML"><record id="job_function_sale_order_action_done" model="queue.job.function">
<field name="model_id" ref="sale.model_sale_order"</field>
<field name="method">action_done</field>
<field name="channel_id" ref="channel_sale" />
<field name="related_action" eval='{"func_name": "custom_related_action"}' />
<field name="retry_pattern" eval="{1: 60, 2: 180, 3: 10, 5: 300}" />
</record></code>
</pre>
<p>The general form for the <code>name</code> is: <code><model.name>.method</code>.</p>
<p>The channel, related action and retry pattern options are optional, they are
documented below.</p>
<p>When writing modules, if 2+ modules add a job function or channel with the same
name (and parent for channels), they'll be merged in the same record, even if
they have different xmlids. On uninstall, the merged record is deleted when all
the modules using it are uninstalled.</p>
<p><strong>Job function: channel</strong></p>
<p>The channel where the job will be delayed. The default channel is <code>root</code>.</p>
<p><strong>Job function: related action</strong></p>
<p>The <em>Related Action</em> appears as a button on the Job's view.
The button will execute the defined action.</p>
<p>The default one is to open the view of the record related to the job (form view
when there is a single record, list view for several records).
In many cases, the default related action is enough and doesn't need
customization, but it can be customized by providing a dictionary on the job
function:</p>
<pre>
<code lang="python">{
"enable": False,
"func_name": "related_action_partner",
"kwargs": {"name": "Partner"},
}</code>
</pre>
<ul class="simple">
<li><code>enable</code>: when <code>False</code>, the button has no effect (default: <code>True</code>)</li>
<li><code>func_name</code>: name of the method on <code>queue.job</code> that returns an action</li>
<li><code>kwargs</code>: extra arguments to pass to the related action method</li>
</ul>
<p>Example of related action code:</p>
<pre>
<code lang="python">class QueueJob(models.Model):
_inherit = 'queue.job'
def related_action_partner(self, name):
self.ensure_one()
model = self.model_name
partner = self.records
action = {
'name': name,
'type': 'ir.actions.act_window',
'res_model': model,
'view_type': 'form',
'view_mode': 'form',
'res_id': partner.id,
}
return action</code>
</pre>
<p><strong>Job function: retry pattern</strong></p>
<p>When a job fails with a retryable error type, it is automatically
retried later. By default, the retry is always 10 minutes later.</p>
<p>A retry pattern can be configured on the job function. What a pattern represents
is "from X tries, postpone to Y seconds". It is expressed as a dictionary where
keys are tries and values are seconds to postpone as integers:</p>
<pre>
<code lang="python">{
1: 10,
5: 20,
10: 30,
15: 300,
}</code>
</pre>
<p>Based on this configuration, we can tell that:</p>
<ul class="simple">
<li>5 first retries are postponed 10 seconds later</li>
<li>retries 5 to 10 postponed 20 seconds later</li>
<li>retries 10 to 15 postponed 30 seconds later</li>
<li>all subsequent retries postponed 5 minutes later</li>
</ul>
<p><strong>Bypass jobs on running Odoo</strong></p>
<p>When you are developing (ie: connector modules) you might want
to bypass the queue job and run your code immediately.</p>
<p>To do so you can set <cite>TEST_QUEUE_JOB_NO_DELAY=1</cite> in your enviroment.</p>
<p><strong>Bypass jobs in tests</strong></p>
<p>When writing tests on job-related methods is always tricky to deal with
delayed recordsets. To make your testing life easier
you can set <cite>test_queue_job_no_delay=True</cite> in the context.</p>
<p>Tip: you can do this at test case level like this</p>
<pre>
<code lang="python">@classmethod
def setUpClass(cls):
super().setUpClass()
cls.env = cls.env(context=dict(
cls.env.context,
test_queue_job_no_delay=True, # no jobs thanks
))</code>
</pre>
<p>Then all your tests execute the job methods synchronously
without delaying any jobs.</p>
<a name="known-issues-roadmap"></a>
<h2><a class="toc-backref" href="#id7">Known issues / Roadmap</a></h2>
<ul class="simple">
<li>After creating a new database or installing <code>queue_job</code> on an
existing database, Odoo must be restarted for the runner to detect it.</li>
<li>When Odoo shuts down normally, it waits for running jobs to finish.
However, when the Odoo server crashes or is otherwise force-stopped,
running jobs are interrupted while the runner has no chance to know
they have been aborted. In such situations, jobs may remain in
<code>started</code> or <code>enqueued</code> state after the Odoo server is halted.
Since the runner has no way to know if they are actually running or
not, and does not know for sure if it is safe to restart the jobs,
it does not attempt to restart them automatically. Such stale jobs
therefore fill the running queue and prevent other jobs to start.
You must therefore requeue them manually, either from the Jobs view,
or by running the following SQL statement <em>before starting Odoo</em>:</li>
</ul>
<pre>
<code lang="sql">update queue_job set state='pending' where state in ('started', 'enqueued')</code>
</pre>
<a name="changelog"></a>
<h2 class="with-subtitle"><a class="toc-backref" href="#id8">Changelog</a></h2>
<h2 class="section-subtitle" id="next"><span class="section-subtitle">Next</span></h2>
<ul class="simple">
<li>[ADD] Run jobrunner as a worker process instead of a thread in the main
process (when running with --workers > 0)</li>
<li>[REF] <code>@job</code> and <code>@related_action</code> deprecated, any method can be delayed,
and configured using <code>queue.job.function</code> records</li>
<li>[MIGRATION] from 13.0 branched at rev. e24ff4b</li>
</ul>
<a name="bug-tracker"></a>
<h2><a class="toc-backref" href="#id9">Bug Tracker</a></h2>
<p>Bugs are tracked on <a class="reference external" href="https://github.com/OCA/queue/issues">GitHub Issues</a>.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us smashing it by providing a detailed and welcomed
<a class="reference external" href="https://github.com/OCA/queue/issues/new?body=module:%20queue_job%0Aversion:%2014.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**">feedback</a>.</p>
<p>Do not contact contributors directly about support or help with technical issues.</p>
<a name="credits"></a>
<h2><a class="toc-backref" href="#id10">Credits</a></h2>
<a name="authors"></a>
<h3><a class="toc-backref" href="#id11">Authors</a></h3>
<ul class="simple">
<li>Camptocamp</li>
<li>ACSONE SA/NV</li>
</ul>
<a name="contributors"></a>
<h3><a class="toc-backref" href="#id12">Contributors</a></h3>
<ul class="simple">
<li>Guewen Baconnier <<a class="reference external" href="mailto:guewen.baconnier@camptocamp.com">guewen.baconnier@camptocamp.com</a>></li>
<li>Stéphane Bidoul <<a class="reference external" href="mailto:stephane.bidoul@acsone.eu">stephane.bidoul@acsone.eu</a>></li>
<li>Matthieu Dietrich <<a class="reference external" href="mailto:matthieu.dietrich@camptocamp.com">matthieu.dietrich@camptocamp.com</a>></li>
<li>Jos De Graeve <<a class="reference external" href="mailto:Jos.DeGraeve@apertoso.be">Jos.DeGraeve@apertoso.be</a>></li>
<li>David Lefever <<a class="reference external" href="mailto:dl@taktik.be">dl@taktik.be</a>></li>
<li>Laurent Mignon <<a class="reference external" href="mailto:laurent.mignon@acsone.eu">laurent.mignon@acsone.eu</a>></li>
<li>Laetitia Gangloff <<a class="reference external" href="mailto:laetitia.gangloff@acsone.eu">laetitia.gangloff@acsone.eu</a>></li>
<li>Cédric Pigeon <<a class="reference external" href="mailto:cedric.pigeon@acsone.eu">cedric.pigeon@acsone.eu</a>></li>
<li>Tatiana Deribina <<a class="reference external" href="mailto:tatiana.deribina@avoin.systems">tatiana.deribina@avoin.systems</a>></li>
<li>Souheil Bejaoui <<a class="reference external" href="mailto:souheil.bejaoui@acsone.eu">souheil.bejaoui@acsone.eu</a>></li>
<li>Eric Antones <<a class="reference external" href="mailto:eantones@nuobit.com">eantones@nuobit.com</a>></li>
</ul>
<a name="maintainers"></a>
<h3><a class="toc-backref" href="#id13">Maintainers</a></h3>
<p>This module is maintained by the OCA.</p>
<a class="reference external image-reference" href="https://odoo-community.org"><img alt="Odoo Community Association" src="https://odoo-community.org/logo.png" /></a>
<p>OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.</p>
<p>Current <a class="reference external" href="https://odoo-community.org/page/maintainer-role">maintainer</a>:</p>
<p><a class="reference external" href="https://github.com/guewen"><img alt="guewen" src="https://github.com/guewen.png?size=40px" /></a></p>
<p>This module is part of the <a class="reference external" href="https://github.com/OCA/queue/tree/14.0/queue_job">OCA/queue</a> project on GitHub.</p>
<p>You are welcome to contribute. To learn how please visit <a class="reference external" href="https://odoo-community.org/page/Contribute">https://odoo-community.org/page/Contribute</a>.</p>