Investigating Paranormal Shell Activities

Recently, while attending a conference, I observed an unusual occurrence in my terminal emulator: terminal tab windows were getting highlighted without any apparent notification within the shell session. Time to unpack our trusty debugging tools to uncover the mystery behind these activities.

Once a tab encountered this issue, it persisted in being highlighted until I closed the affected tab and opened a new one. Another peculiar symptom manifested when certain terminal applications, running within the affected (or possibly infected?) terminal, displayed strange sequences of characters like ^I^O^I^O. Given that the conference I was attending focused on security, my heightened awareness prompted me to investigate the root cause of this issue. Upon returning to my hotel room, I delved further into the matter, particularly after noticing that additional tabs were affected by the highlighting problem.

Hunting for Ghosts

The initial step involved inspecting a tab affected by the issue. Despite examining tab-specific settings in the terminal emulator, no alarming findings emerged. Consequently, I turned my attention to the shell itself and employed the strace system call tracer¹. The objective was to investigate whether the highlighting originated from keyboard events or similar triggers, entertaining the possibility that my laptop keyboard might be malfunctioning.

Tracing read() and write() syscalls promptly revealed that my shell was indeed receiving input without any corresponding keystrokes from my end. Specifically, the shell was consistently receiving three characters: \033 (hex 0x1b), [, followed by either I or O. Subsequently, the shell emitted another character, consistently \7.

read(0, "\33", 1)                       = 1
read(0, "[", 1)                         = 1
read(0, "I", 1)                         = 1
write(2, "\7", 1)                       = 1

As I examined the strace output further, a realization struck me - 0x07 corresponds to the ASCII bell character (BEL²). This discovery literally rang a bell inside my head. It explained why the terminal emulator highlighted the tab, aligning perfectly with the expected behavior when the shell writes BEL to the output stream.

However, 0x1b, [ and I or O still puzzled me. I was already aware that octal 033 represents the escape character³. These days, such sequences are commonly used in the shell PS1 variable for creating visually appealing terminals. Although, in my case, it wasn’t related to color formatting. My attempts to search for the three-byte sequence online proved challenging due to its non-google-friendly nature, with multiple possible representations. Subsequent searches involving only the escape character and “KDE Konsole” my terminal emulator, led me to the Konsole source code on GitHub. Trusting my instincts, I delved into the Konsole source code, specifically searching for [I and [O. This search led me to a C++ method called Vt102Emulation::focusChanged⁴.

void Vt102Emulation::focusChanged(bool focused)
{
    if (_reportFocusEvents) {
        sendString(focused ? "\033[I" : "\033[O");
    }
}

It became clear that on every tab focus change, Konsole’s code was triggering the peculiar sequence. Specifically, on each tab focus change, Konsole sends either \033[I or \033[O to the shell. So far, so good. However, the mystery remained as to why I only observed this behavior on certain tabs.

Further exploration of Konsole’s source code revealed that sending the three-byte sequence is governed by the variable _reportFocusEvents, and this variable, in turn, is influenced by control sequence \033[?1004h resp. \033[?1004l. Indeed, executing the command echo -ne "\033[?1004l in an affected terminal instantaneously cured the issue. Reading these sequences is sometimes not obvious, a control sequence typically consists of an ASCII ESC character, a CSI (Control Sequence Introducer, [ in our case), and parameters. These days they are mostly used for colorful⁵ output in the shell. Enabling focus events is defined by XTerm⁶ as CSI ? 1004 h and disabling as CSI ? 1004 h, KDE Konsole follows this definition.

The question still lingers: why aren’t all tabs affected, and why didn’t I notice the problem earlier? It dawned on me that there might be a program enabling focus reporting on the affected tabs. Upon further reflection, I realized that the issue occurred exclusively on tabs I used to ssh into a particular server. Considering my unreliable internet connection leading to frequent ssh timeouts and remote shell terminations, it occurred to me: What if some program on the remote server is enabling focus reporting and leaving it enabled?

An examination of my bash history revealed that I primarily used git, vim, qemu, and other fundamental shell tools - nothing out of the ordinary. To delve deeper, I conducted additional research to determine if any of these tools, git, vim, or qemu made use of focus reporting. As it turned out, the newer version of vim, version 9, indeed includes support for focus events⁷. It suddenly struck me that I had recently upgraded to this latest version on the remote server.

To verify the hypothesis, I have created the following bpftrace⁸ script:

tracepoint:syscalls:sys_enter_write
/uid == 1000 && args->fd == 1 && args->count < 64/
{
        if (strcontains(str(args->buf, args->count), "\033[?1004h")) {
                printf("process %s (%d) enabled focus events\n", comm, pid);
        } else if (strcontains(str(args->buf, args->count), "\033[?1004l")) {
                printf("process %s (%d) disabled focus events\n", comm, pid);
        }
}

It checks whether a program run by me (uid 1000) writes the focus events enable or disable control sequence to stdout.

Indeed, vim enables these events upon start and disables them later upon regular exit.

$ bpftrace csi_write.bt
Attaching 1 probe...
process vi (18811) enabled focus events
process vi (18811) disabled focus events

From now on the following two lines are part of my .vimrc:

set t_fd=
set t_fe=

They make sure to have vim’s focus reporting feature disabled: I don’t need it.

Summary

Whenever a vim session was interrupted due to a connection timeout on this particular server, it left focus reporting enabled on my laptop’s shell. This, in turn, caused the shell to emit bell events, and explained why I occasionally observed ^I^O when running programs that didn’t consume any input. The recurrent issues with my internet connection played a crucial role in triggering this situation frequently. The interruptions in the connection led to instances where vim on the remote server was abruptly terminated, failing to disable focus reporting in the process.

Indeed, there were no ghosts in my shell - just a case of me getting a bit rusty with remembering shell control sequences, coupled with a problematic internet connection and an upgraded vim.