sched/fair: Do not re-read ->h_load_next during hierarchical load calculation [Linux 3.16.72]

This Linux kernel change "sched/fair: Do not re-read ->h_load_next during hierarchical load calculation" is included in the Linux 3.16.72 release. This change is authored by Mel Gorman <mgorman [at]> on Tue Mar 19 12:36:10 2019 +0000. The commit for this change in Linux stable tree is bc05cf8 (patch) which is from upstream commit 0e9f024. The same Linux upstream change may have been applied to various maintained Linux releases and you can find all Linux releases containing changes from upstream 0e9f024.

sched/fair: Do not re-read ->h_load_next during hierarchical load calculation

commit 0e9f02450da07fc7b1346c8c32c771555173e397 upstream.

A NULL pointer dereference bug was reported on a distribution kernel but
the same issue should be present on mainline kernel. It occured on s390
but should not be arch-specific.  A partial oops looks like:

  Unable to handle kernel pointer dereference in virtual kernel address space
  Call Trace:
    vhost_poll_wakeup+0x3a/0x50 [vhost]

The bug hits any time between 1 hour to 3 days. The dereference occurs
in update_cfs_rq_h_load when accumulating h_load. The problem is that
cfq_rq->h_load_next is not protected by any locking and can be updated
by parallel calls to task_h_load. Depending on the compiler, code may be
generated that re-reads cfq_rq->h_load_next after the check for NULL and
then oops when reading se->avg.load_avg. The dissassembly showed that it
was possible to reread h_load_next after the check for NULL.

While this does not appear to be an issue for later compilers, it's still
an accident if the correct code is generated. Full locking in this path
would have high overhead so this patch uses READ_ONCE to read h_load_next
only once and check for NULL before dereferencing. It was confirmed that
there were no further oops after 10 days of testing.

As Peter pointed out, it is also necessary to use WRITE_ONCE() to avoid any
potential problems with store tearing.

Signed-off-by: Mel Gorman <>
Signed-off-by: Peter Zijlstra (Intel) <>
Reviewed-by: Valentin Schneider <>
Cc: Linus Torvalds <>
Cc: Mike Galbraith <>
Cc: Peter Zijlstra <>
Cc: Thomas Gleixner <>
Fixes: 685207963be9 ("sched: Move h_load calculation to task_h_load()")
Signed-off-by: Ingo Molnar <>
[bwh: Backported to 3.16: use ACCESS_ONCE()]
Signed-off-by: Ben Hutchings <>

There are 6 lines of Linux source code added/deleted in this change. Code changes to Linux kernel are as follows.

 kernel/sched/fair.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index 3284106..08b6858 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -5487,10 +5487,10 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
    if (cfs_rq->last_h_load_update == now)

-   cfs_rq->h_load_next = NULL;
+   ACCESS_ONCE(cfs_rq->h_load_next) = NULL;
    for_each_sched_entity(se) {
        cfs_rq = cfs_rq_of(se);
-       cfs_rq->h_load_next = se;
+       ACCESS_ONCE(cfs_rq->h_load_next) = se;
        if (cfs_rq->last_h_load_update == now)
@@ -5500,7 +5500,7 @@ static void update_cfs_rq_h_load(struct cfs_rq *cfs_rq)
        cfs_rq->last_h_load_update = now;

-   while ((se = cfs_rq->h_load_next) != NULL) {
+   while ((se = ACCESS_ONCE(cfs_rq->h_load_next)) != NULL) {
        load = cfs_rq->h_load;
        load = div64_ul(load * se->avg.load_avg_contrib,
                cfs_rq->runnable_load_avg + 1);

Leave a Reply

Your email address will not be published. Required fields are marked *