FoxQuirk 🦊 @quirk@computerfairi.es

For those who don't know, Terminfo is a system to document terminals and their quirks, what features they support.

This is because telling a terminal to do something usually involves sending it an escape sequence, and if it misunderstands that it might do something wrong, resulting in broken output/input.

This seems like something that should exist, but unfortunately, terminfo is:

• incomplete
• terribly documented and terrible to use
• updates too slowly
• sometimes takes literal decades to add new capabilities, only after someone important asked nicely
• fails terribly if an entry does not exist
• has no fallback mechanism
• has no versioning
• constructed to excert maximum pressure on terminals to claim to be xterm
• excerts too little pressure on terminals to not have useless differences
• used inconsistently even by its own authors

In short: Using terminfo is a bad experience, and terminfo has caused terminals to both diverge more in uninteresting ways while simultaneously making good new features harder to spread. I have come to believe that it is a net-negative on the terminal ecosystem.

This is going to be a rant. Note that I will be referring to "the terminfo authors" throughout, but I want to make it clear I have nothing specifically against these people. I won't name anyone specifically. They have done a lot of work that, at the time, was extremely important and helpful. My point is that it no longer is. At this point, terminfo is a net-negative and adhering to it is holding us back. I am ranting against this piece of software, not the people.

I will also not spend a lot of time delineating terminfo and ncurses - they're made by the same authors, and what ncurses does is how things interpret terminfo. I don't believe it is very important.

I will also entirely disregard alternative implementations - ncurses terminfo is the one that matters, and most of this is about the concepts and not the specific database. It is also about defaults.

I am also quite uninterested in assigning blame. If terminfo is frequently misused that is a problem with the state of things - and while it is often because terminfo is designed so that misusing it is easy and tempting, I am uninterested in legislating who was at fault for which exact problem.

Here's how terminfo is supposed to work: Your terminal sets the $TERM variable to indicate which terminal it is, your program queries the terminfo database which looks for a file on your system that includes the actual capabilities for that name. Capabilities include things like "can do 8 colors" or "sends this sequence to mean up-arrow" or "understands this sequence to clear the screen".

These files, which I call the "terminfo entries", form the terminfo database, which is usually distributed as part of the system, which means in our world of long term support distros it is often years out of date. Especially since people, you know, ssh onto remote servers that run old-software-by-design systems like Ubuntu LTS or even RHEL. Some may even keep some terminfo files in a "terminfo-extra" package to save on precious kilobytes of disk space. So if you aren't root and connect to a server that doesn't have that package? If you're lucky, you'll just not have niceties like color.

An extra terrible example is Apple, which makes a terminal ("Terminal.app") that had italics support in at least 2017. As far as I know their own terminfo, on their own operating system, did not claim italics support in 2022. In 2017 it shipped a terminfo from 2013.

So if you make a new terminal, you either need to wait for years until your terminfo entry is incorporated into the database, and years more until it reaches users, and years more until it reaches servers. Repeat this every time you add a new feature.

And the failure mode for this is that your terminal is not recognized at all because it never grew a fallback mechanism. There is no way to say "hey I'm bterm, but if you don't know that just treat me like xterm". There is also no versioning mechanism ("if this is konsole > 20.04 it supports cursor shapes"), it wants the terminfo database to be in sync with the terminal version. Which, again... SERVERS EXIST. This would have been easy to do in a backwards-compatible manner - add a "$TERM_REAL" variable that points to the real terminfo entry, and if you can't find it keep using $TERM. Make a base entry that documents the consensus among terminals still in development and have them use that.

Seriously, set $TERM to something not on your system and try to start emacs or nano - they will REFUSE TO START. Vim will print a stern error and run in ansi-compatibility mode, as if you're in 1989.

There is also software that hardcodes specific $TERM values and sidesteps terminfo lookups. Which of course the terminfo people don't like, and will tell you is wrong, but it is reality.

All this combines to create maximum pressure for terminals to claim they are "xterm-256color". Some terminals resist, and instead create mechanisms to push their terminfo elsewhere, or document how to get their terminfo entry. This is brittle and should not be necessary.

And the terminfo authors are quite cross about anyone using their xterm(-256color) entry, and instead create fictional entries for users to be confused by. E.g. KDE's konsole sets $TERM to xterm-256color. Has for as long as I've known about this. Terminfo ships a "konsole-256color" entry that the terminfo people want you to use instead, but it's old and outdated. For instance it claims that konsole does not know the "E3" ("clear scrollback") capability. This is outright wrong. Using these entries gets you one step forward, two steps back.

I am so confused that they seem to recognize the problem of using a rare terminfo entry and just shrug. It's written so the maximally competent user can build their own terminfo database and point programs there, and repeat that on every server they connect to. And that's a kind of making software that I personally vehemently disagree with. I want things to work by default instead of work for the twelve people who've studied them in detail and have limitless time to fiddle.

And in terms of the capabilities that are there, it is extremely slow to add them. It took literal decades to add RGB-color support to terminfo (it was added in ncurses 6.1 in 2018).

Bracketed paste was only added about 10 years after it became almost ubiquitous when Bram "Vim" Moolenaar asked nicely. This was 8 years after bash added support, 20 years after xterm gained it and 17 after xterm documented it. Keep in mind that xterm is developed by the terminfo maintainer.

And when it does add these things, it adds them with maximum flexibility - it added bracketed paste as two capabilities "BD" and "BE", which are to be set to escape sequences. Instead of, you know, just adding booleans so at least the fucking sequence stays the same. We need maximum allowance for terminals to diverge in useless ways.

Except actually no. You can change the bracketed paste enable and disable sequences, but the sequences the terminal sends you to start and end pasted text aren't actually included in this, and there's no way to express semantics so the terminal can't tell you how it treats bracketed-paste-sequences inside of pasted text (will it strip them? double them? no way to tell except for adding a new capability!).

It is also inflexible in face of configuration. There is no capability to tell you if a terminal treats ambiguous width codepoints as wide or narrow, but even if there were these are often configurable, and because terminfo needs a file on disk the only way to reflect that would be to change $TERM once you do that.

Terminfo is written for a world where companies build proprietary terminals and determine the escape sequences by rolls of the dice, but release them extremely slowly and once, calling new revisions by new names. It is made for a world with 64 F-keys. It is not made for a world where terminals are software that is developed and gains new features, sometimes even innovating entirely new things. It is great if you want to know what a Data General Dasher D200 did in 1984.

And yes, you've seen that "BD" and "BE" and "E3" up there? Those are the capability names, and they're emblematic of terminfo's real problem with usability and documentation.

The way this works is that any capability has three names: The boolean name you'd use in C-code if you're using ncurses, the "terminfo" code and the "termcap" code - termcap was a different terminal capabilities database that thankfully is dead now (even the BSDs ship terminfo).

The boolean capability names are okay-ish, but the other two are maximally unhelpful - "exit_attribute_mode", which resets all styling, is "sgr0" or "me".

I've never been able to actually find good documentation on any of this. The ncurses/terminfo/xterm website features a lot of talk, but much of it is conversational, explaining the history of things.

There's not a lot of "hey, here's what this capability does" type of documentation. You kinda have to learn by osmosis.

I have searched for what "BD" and "BE" and "E3" stand for and have found nothing. I guess it's "BracketeD paste" and "Bracketed pastE" and "clEar scrollback3"?

What you do find is statements like "Color came into the picture fairly late in the history of terminals", which kind of ignores that we have now had over FORTY YEARS of additional history since color came into the picture. Seriously, color is mentioned in ECMA-48 second edition from literally 1979.

And even the terminfo authors don't use it consistently. tput clear, which is part of the ncurses package, made by the author of xterm and keeper of the sacred terminfo database, will not just use the "clear" sequence. It will use clear and "E3" (clear scrollback). Which boggles my mind because I find those two completely different operations and the second is destructive!

And as we've seen above, even the people who could use it to fully document their terminal's capabilities don't actually do it. I have no idea how they expect anyone to use bracketed paste in a portable manner after they've failed to add it for SEVENTEEN YEARS. We need maximum allowance for terminals to be special and for people to dive into their terminal and write software that only works properly on that terminal, or to have to detect it in some other manner, working around terminfo.

In a way, terminfo ensures maximum uselessness.

Terminals also like adding new features, and sometimes, when they do use their own terminfo, they even think of adding a capability. Only because these are barely documented anywhere terminals that follow suit on supporting the feature forget to include the capability.

For instance both rio and wezterm support the kitty keyboard protocol, which kitty's terminfo entry has the "fullkbd" capability for. Neither rio nor wezterm advertise it in their respective terminfo entries (in their current releases). The best documentation for that capability I can find is a single line comment in the code. This isn't specific to kitty, most terminfo extensions have that level of documentation.

And if the capability isn't actually used consistently to advertise support, it's useless. You can just do a string comparison for "$TERM = xterm-kitty" instead of checking "fullkbd" and would be exactly as wrong.

The only way these capabilities are useful is if you can rely on them to be there when the terminal has a specific feature, and rely on them to be absent when it doesn't. Neither is the case. For many features, no capability exists at all.

ncurses deserves special mention here, because it is a C library that likes setting things to null pointers if they aren't set. It also does macro magic to have some variables behind another pointer, so you need to check cur_term even if you only want enter_attribute_mode. So it is quite easy to write a program that uses terminfo via ncurses that only crashes on some systems, depending on the value of $TERM.

In short: Terminfo, at this point in time, is unhelpful. It has both false-positives and false-negatives, it has terrible modes of failure, it is hard to use and understand. It is neither necessary nor sufficient.

It was invented in a very different time and it shows. Now, in the world we live in, terminfo is useless.

@EricAlper

@paninid

...and accurate to the book, too.

When I read it for the first time, I was surprised by how much of a 19th-Century technothriller it was. In the book, Jonathan Harker has a Kodak camera. There are lots and lots of telegrams-- tweets. Most of the book was written as short epistles -- blog posts.

YOU ARE A TRUE WARRIOR. I AM PROUD OF YOU.