How do system programmers think?

What else can I do? [to avoid runtime errors] How do people who write major system programs think about this kind of problems?

Unit tests.

Conceptually the specify example you give could theoretically be amenable to static checks on your code, ie a better compiler.

The other common way that you already address, is to wrap the problematic code ensuring that the exception is handled, avoided or otherwise would generate a compile time error. ie perhaps with a builder class.

C is a fairly low level language so its lacking in some of the compile time abstractions that haskel or an OOP language has.

But in general, You can just add a unit/integration test to check that no error is thrown at runtime for given sets of parameters.

You can limit the range of these parameters by wrapping the direct function calls in units which restrict the ways in which they can be used and only calling and testing those units rather than the underlying functions.

For instance with your example: (excuse my pseudo code)

getlayersurface takes the problematic parameter surface object which must be uninitialized. But we could wrap the call in a function which instead takes a new class: wlsurfaceuninitialised written by ourselves.

This class would have its own methods to attach/commit a buffer which would return a wlsurfaceinitialised object.

wrap all this up in some immutability, add unit tests to make sure it works and you now have the compile time checks you were looking for.

Since the underlying system is, practically, not reliable.

Whilst there are unreliable systems, it doesn't sound like you are interacting with that much unreliability. What you are interacting with is complexity. The job of a programmer is to manage complexity.

So, formally speaking, I must make a judgement on an infinite language of execution traces. This is again an impossible task.

You will have to categorize the infinite language of execution traces into a finite set of categories. Whilst there are infinitely many ways of doing this categorization, you will have to pick one that fits your requirements.

Enumerating all the relevant conditions and precisely understanding what they mean is an impossible task.

It's not impossible. It may be lengthy, but for every system I have interacted with there is a finite amount of documentation.

If what you observe isn't covered by documentation, record what you did to make what you observed. That is a start to making your own documentation.

In almost all projects, you don't need to do this all up front. Unless your project is safety critical, you can try something, see if it meets you requirements, and tweak that.

So, tldr, real-world programming is hard? (I'm the upvoter btw, not the downvoter.)

The reality of computer programming is not to produce something that works correctly for the life of the universe, but to assist human activity for the time being and to "save labour", and do so with enough reliability to be useful.

Haskell programs may be guaranteed to run in their own terms, but the hardware those programs drive is not guaranteed to work correctly, and the program itself is not guaranteed to be correct (in the sense of doing what it should, or striking a reasonable balance in the circumstances between competing and irreconcilable goals).

You've clearly acquired involvement in programming via academia (hence the use of Haskell), where there has been decades of lack of common sense about what real-world programming is about.

You might therefore have found that you are talented in ways that won't make you a good industrial programmer, or you might be perfectly capable of being a good industrial programmer but are about to find that an academic course of study (and association with academics) has conveyed you very little relevant experience and competency and has in fact twisted your concepts and expectations, and you are now about to start again.

There is no simple way of boiling down a professional area of practice, but amongst various things that professionals do do, is managing the complexity of solutions so as not to exhaust the available budget of programmer labour available (i.e. your labour), and controlling projects which (when kept within a budget) will work too unreliably to be useful (computers can be unreliable either because they produce wrong results too often or with too great consequences, or because they stop processing and require intervention and alteration too frequently or urgently).

Doing these tasks skilfully is what the job consists of.

There's obviously lots more that can be said, but obviously one of the main mistakes you're making is assuming your programs should be truly perfect and that programming computers will be easy, and worrying that this is not the case. Experienced computer programmers don't think like that.

EDIT:

On the points added to the question, the problem is that a "crash" is only conventional in definition. You can avoid a "crash" simply by removing or swallowing all the exceptions from your program.

Proving that a program "terminates" is rarely useful. Some programs are supposed to run indefinitely. Others must not merely terminate before the death of the Sun, but within an acceptable budget of time.

As for reducing bugs by decomposition, the problem is that you can solve every problem with another independent component, except the problem of too many independent components.

And the vast majority of useful programs have to be globally stateful, respond to external events, and evolve their states in ways that are not fully determined at the outset of operation.

What you're mentioning are the common canards peddled by academics about functional languages like "global state", "side effect" data flows, and so on, and experienced practioners know (to be blunt) it's all a load of nonsense!

Professional practice is about working competently with what they dub "global state" and "side effects", including managing the complexity of these and moderating their usage to what is reasonably essential (rather than eliminating them on the theory they are inessential).