perf/ring_buffer: Add ordering to rb->nest increment [Linux 4.14.129]

perf/ring_buffer: Add ordering to rb->nest increment [Linux 4.14.129]

This Linux kernel change "perf/ring_buffer: Add ordering to rb->nest increment" is included in the Linux 4.14.129 release. This change is authored by Peter Zijlstra <peterz [at]> on Fri May 17 13:52:32 2019 +0200. The commit for this change in Linux stable tree is 2bc92a4 (patch) which is from upstream commit 3f9fbe9. The same Linux upstream change may have been applied to various maintained Linux releases and you can find all Linux releases containing changes from upstream 3f9fbe9.

perf/ring_buffer: Add ordering to rb->nest increment

[ Upstream commit 3f9fbe9bd86c534eba2faf5d840fd44c6049f50e ]

Similar to how decrementing rb->next too early can cause data_head to
(temporarily) be observed to go backward, so too can this happen when
we increment too late.

This barrier() ensures the rb->head load happens after the increment,
both the one in the 'goto again' path, as the one from
perf_output_get_handle() -- albeit very unlikely to matter for the

Suggested-by: Yabin Cui <[email protected]>
Signed-off-by: Peter Zijlstra (Intel) <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Linus Torvalds <[email protected]>
Cc: Peter Zijlstra <[email protected]>
Cc: Stephane Eranian <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Vince Weaver <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Fixes: ef60777c9abd ("perf: Optimize the perf_output() path by removing IRQ-disables")
Link:[email protected]
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Sasha Levin <[email protected]>

There are 9 lines of Linux source code added/deleted in this change. Code changes to Linux kernel are as follows.

 kernel/events/ring_buffer.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index fde8532..aef2af8 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -49,6 +49,15 @@ static void perf_output_put_handle(struct perf_output_handle *handle)
    unsigned long head;

+   /*
+    * In order to avoid publishing a head value that goes backwards,
+    * we must ensure the load of @rb->head happens after we've
+    * incremented @rb->nest.
+    *
+    * Otherwise we can observe a @rb->head value before one published
+    * by an IRQ/NMI happening between the load and the increment.
+    */
+   barrier();
    head = local_read(&rb->head);


Leave a Reply

Your email address will not be published. Required fields are marked *