mm: memcontrol: use per-cpu stocks for socket memory uncharging

This change “mm: memcontrol: use per-cpu stocks for socket memory uncharging” in Linux kernel is authored by Roman Gushchin <guro [at]> on Fri Sep 8 16:13:09 2017 -0700.

mm: memcontrol: use per-cpu stocks for socket memory uncharging

We've noticed a quite noticeable performance overhead on some hosts with
significant network traffic when socket memory accounting is enabled.

Perf top shows that socket memory uncharging path is hot:
  2.13%  [kernel]                [k] page_counter_cancel
  1.14%  [kernel]                [k] __sk_mem_reduce_allocated
  1.14%  [kernel]                [k] _raw_spin_lock
  0.87%  [kernel]                [k] _raw_spin_lock_irqsave
  0.84%  [kernel]                [k] tcp_ack
  0.84%  [kernel]                [k] ixgbe_poll
  0.83%  < workload >
  0.82%  [kernel]                [k] enqueue_entity
  0.68%  [kernel]                [k] __fget
  0.68%  [kernel]                [k] tcp_delack_timer_handler
  0.67%  [kernel]                [k] __schedule
  0.60%  < workload >
  0.59%  [kernel]                [k] __inet6_lookup_established
  0.55%  [kernel]                [k] __switch_to
  0.55%  [kernel]                [k] menu_select
  0.54%            [.] __memcpy_avx_unaligned

To address this issue, the existing per-cpu stock infrastructure can be

refill_stock() can be called from mem_cgroup_uncharge_skmem() to move
charge to a per-cpu stock instead of calling atomic

To prevent the uncontrolled growth of per-cpu stocks, refill_stock()
will explicitly drain the cached charge, if the cached value exceeds

This allows significantly optimize the load:
  1.21%  [kernel]                [k] _raw_spin_lock
  1.01%  [kernel]                [k] ixgbe_poll
  0.92%  [kernel]                [k] _raw_spin_lock_irqsave
  0.90%  [kernel]                [k] enqueue_entity
  0.86%  [kernel]                [k] tcp_ack
  0.85%  < workload >
  0.74%          [.] 0x000000000061bf24
  0.73%  [kernel]                [k] __schedule
  0.67%  [kernel]                [k] __fget
  0.63%  [kernel]                [k] __inet6_lookup_established
  0.62%  [kernel]                [k] menu_select
  0.59%  < workload >
  0.59%  [kernel]                [k] __switch_to
  0.57%            [.] __memcpy_avx_unaligned

Signed-off-by: Roman Gushchin <>
Acked-by: Johannes Weiner <>
Acked-by: Michal Hocko <>
Cc: Vladimir Davydov <>
Signed-off-by: Andrew Morton <>
Signed-off-by: Linus Torvalds <>

This Linux change may have been applied to various maintained Linux releases and you can find Linux releases including commit 475d048.

There are 6 lines of Linux source code added/deleted in this change. Code changes to Linux kernel are as follows.

 mm/memcontrol.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 126a939b..ca83f38 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1792,6 +1792,9 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
 	stock->nr_pages += nr_pages;
+	if (stock->nr_pages > CHARGE_BATCH)
+		drain_stock(stock);
@@ -5886,8 +5889,7 @@ void mem_cgroup_uncharge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages)
 	this_cpu_sub(memcg->stat->count[MEMCG_SOCK], nr_pages);
-	page_counter_uncharge(&memcg->memory, nr_pages);
-	css_put_many(&memcg->css, nr_pages);
+	refill_stock(memcg, nr_pages);
 static int __init cgroup_memory(char *s)

The commit for this change in Linux stable tree is 475d048 (patch).

Leave a Reply

Your email address will not be published. Required fields are marked *