This Linux kernel change "x86/asm/entry/64: Disentangle error_entry/exit gsbase/ebx/usermode code" is included in the Linux 3.16.72 release. This change is authored by Andy Lutomirski <luto [at] kernel.org> on Tue Jun 9 12:36:01 2015 -0700. The commit for this change in Linux stable tree is bb0737d (patch) which is from upstream commit 539f511. The same Linux upstream change may have been applied to various maintained Linux releases and you can find all Linux releases containing changes from upstream 539f511.
x86/asm/entry/64: Disentangle error_entry/exit gsbase/ebx/usermode code commit 539f5113650068ba221197f190267ab727296ef5 upstream. The error_entry/error_exit code to handle gsbase and whether we return to user mdoe was a mess: - error_sti was misnamed. In particular, it did not enable interrupts. - Error handling for gs_change was hopelessly tangled the normal usermode path. Separate it out. This saves a branch in normal entries from kernel mode. - The comments were bad. Fix it up. As a nice side effect, there's now a code path that happens on error entries from user mode. We'll use it soon. Signed-off-by: Andy Lutomirski <[email protected]> Cc: Andrew Morton <[email protected]> Cc: Andy Lutomirski <[email protected]> Cc: Borislav Petkov <[email protected]> Cc: Brian Gerst <[email protected]> Cc: Denys Vlasenko <[email protected]> Cc: H. Peter Anvin <[email protected]> Cc: Linus Torvalds <[email protected]> Cc: Peter Zijlstra <[email protected]> Cc: Thomas Gleixner <[email protected]> Link: http://lkml.kernel.org/r/[email protected]nel.org [ Prettified it, clarified comments some more. ] Signed-off-by: Ingo Molnar <[email protected]> [bwh: Backported to 3.16 as dependency of commit 18ec54fdd6d1 "x86/speculation: Prepare entry code for Spectre v1 swapgs mitigations": - We do not use %ebx as a flag since we already have a backport of commit b3681dd548d0 "x86/entry/64: Remove %ebx handling from error_entry/exit", so don't add the comments about that - Adjust filename, context] Signed-off-by: Ben Hutchings <[email protected]>
There are 28 lines of Linux source code added/deleted in this change. Code changes to Linux kernel are as follows.
arch/x86/kernel/entry_64.S | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/x86/kernel/entry_64.S b/arch/x86/kernel/entry_64.S index 6e5feb3..ddee7b2 100644 --- a/arch/x86/kernel/entry_64.S +++ b/arch/x86/kernel/entry_64.S @@ -1446,9 +1446,11 @@ ENTRY(error_entry) SWITCH_KERNEL_CR3 testl $3,CS+8(%rsp) je error_kernelspace -error_swapgs: + + /* We entered from user mode */ SWAPGS -error_sti: + +error_entry_done: TRACE_IRQS_OFF ret @@ -1466,8 +1468,15 @@ error_kernelspace: cmpq %rax,RIP+8(%rsp) je bstep_iret cmpq $gs_change,RIP+8(%rsp) - je error_swapgs - jmp error_sti + jne error_entry_done + + /* + * hack: gs_change can fail with user gsbase. If this happens, fix up + * gsbase and proceed. We'll fix up the exception and land in + * gs_change's error handler with kernel gsbase. + */ + SWAPGS + jmp error_entry_done bstep_iret: /* Fix truncated RIP */ @@ -1475,11 +1484,20 @@ bstep_iret: /* fall through */ error_bad_iret: + /* + * We came from an IRET to user mode, so we have user gsbase. + * Switch to kernel gsbase: + */ SWAPGS + + /* + * Pretend that the exception came from user mode: set up pt_regs + * as if we faulted immediately after IRET. + */ mov %rsp,%rdi call fixup_bad_iret mov %rax,%rsp - jmp error_sti + jmp error_entry_done CFI_ENDPROC END(error_entry)