mm/mmu_notifier: use hlist_add_head_rcu() [Linux 4.9.187]

This Linux kernel change "mm/mmu_notifier: use hlist_add_head_rcu()" is included in the Linux 4.9.187 release. This change is authored by Jean-Philippe Brucker <jean-philippe.brucker [at] arm.com> on Thu Jul 11 20:58:50 2019 -0700. The commit for this change in Linux stable tree is 484354b (patch) which is from upstream commit 543bdb2. The same Linux upstream change may have been applied to various maintained Linux releases and you can find all Linux releases containing changes from upstream 543bdb2.

mm/mmu_notifier: use hlist_add_head_rcu()

[ Upstream commit 543bdb2d825fe2400d6e951f1786d92139a16931 ]

Make mmu_notifier_register() safer by issuing a memory barrier before
registering a new notifier.  This fixes a theoretical bug on weakly
ordered CPUs.  For example, take this simplified use of notifiers by a
driver:

    my_struct->mn.ops = &my_ops; /* (1) */
    mmu_notifier_register(&my_struct->mn, mm)
        ...
        hlist_add_head(&mn->hlist, &mm->mmu_notifiers); /* (2) */
        ...

Once mmu_notifier_register() releases the mm locks, another thread can
invalidate a range:

    mmu_notifier_invalidate_range()
        ...
        hlist_for_each_entry_rcu(mn, &mm->mmu_notifiers, hlist) {
            if (mn->ops->invalidate_range)

The read side relies on the data dependency between mn and ops to ensure
that the pointer is properly initialized.  But the write side doesn't have
any dependency between (1) and (2), so they could be reordered and the
readers could dereference an invalid mn->ops.  mmu_notifier_register()
does take all the mm locks before adding to the hlist, but those have
acquire semantics which isn't sufficient.

By calling hlist_add_head_rcu() instead of hlist_add_head() we update the
hlist using a store-release, ensuring that readers see prior
initialization of my_struct.  This situation is better illustated by
litmus test MP+onceassign+derefonce.

Link: http://lkml.kernel.org/r/[email protected]
Fixes: cddb8a5c14aa ("mmu-notifiers: core")
Signed-off-by: Jean-Philippe Brucker <[email protected]>
Cc: Jérôme Glisse <[email protected]>
Cc: Michal Hocko <[email protected]>
Signed-off-by: Andrew Morton <[email protected]>
Signed-off-by: Linus Torvalds <[email protected]>
Signed-off-by: Sasha Levin <[email protected]>

There are 2 lines of Linux source code added/deleted in this change. Code changes to Linux kernel are as follows.

 mm/mmu_notifier.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c
index f4259e4..7a66e37 100644
--- a/mm/mmu_notifier.c
+++ b/mm/mmu_notifier.c
@@ -286,7 +286,7 @@ static int do_mmu_notifier_register(struct mmu_notifier *mn,
     * thanks to mm_take_all_locks().
     */
    spin_lock(&mm->mmu_notifier_mm->lock);
-   hlist_add_head(&mn->hlist, &mm->mmu_notifier_mm->list);
+   hlist_add_head_rcu(&mn->hlist, &mm->mmu_notifier_mm->list);
    spin_unlock(&mm->mmu_notifier_mm->lock);

    mm_drop_all_locks(mm);

Leave a Reply

Your email address will not be published. Required fields are marked *