One of our tests has started failing a bit too often, but only when run as part of a larger test suite. So I've run the test case manually several times, for both ARM and X86. It always passes. But yesterday I got too annoyed by this, and started digging. Unfortunately the test case is the first one in its set, so it's not a matter of it starting in an undefined state. However, that means rerunning the test suite triggers the failure quickly.
The test case basically consists of starting the application, sending it some data, and see what comes out. However, according to the log files, no data was actually received, so there was no output. It showed a connection being made, which was immediately closed again.
The application has a little function that checks if the socket has been closed from the other side, by first calling poll() with a zero timeout, and if that says some data is available, it tries to read one byte. If that returns 0, the socket has been closed. But surely that must be a mistake, in this case. Is there a missing check for when errno is EAGAIN or something, an incorrectly handled timeout, or something? But no, no matter what I modified, the socket was definitely closed. The pair "you have data" plus "read returns 0" does indeed mean that the socket is closed.
The program used for sending the data, in this case, is netcat. It's part of most Linux distributions. Maybe it did something odd? Ok, let's run it under strace, to see if it actually sends anything. Running... and nothing. No output from it whatsoever. Huh? Oh. Everything here is running within Docker, and in this case in X86 mode on an ARM machine, so everything is emulated.
Then I found the netcat option "-v". This made it print something about "Preposterous fd value 18". What's wrong with 18? Everything above -1 is fine, stop complaining! Then I found a change from 2010 or something, increasing the max allowed fd number from 16 to 1024. This is still idiocy, but at least it would allow fd to be 18. The change was for version 1.10.30, and I was running 1.10.41. I checked which file descriptors were open before starting netcat, and now things started to make more sense. When the test was run as part of a suite, there was an extra level or two of shell scripts, and all of them required 2-3 processes as they all needed the rosetta emulator. I closed one of the descriptors under 16 before running netcat, and now the test passed every time. So, it seems Ubuntu had forgotten to include that change in their version. The BSD version doesn't have this check at all, but I didn't quite feel like trying to download and build that, all for a single test case with a functional workaround.
Netcat, in its infinite wisdom, had decided if the file descriptor value was "too high", it immediately closed it and gave up. No meaningful error message (unless run with "-v"), no nothing. Just "nah, don't wanna". File descriptors should be checked that they are above -1, that's all. They can, and will, be just about anything. Most likely relatively small, to make select() fast, but that's it. The newer poll() doesn't care how large the values are.
It may be a good idea to set FD_CLOEXEC on your own file descriptors, to avoid leaking them to child processes.
The test case basically consists of starting the application, sending it some data, and see what comes out. However, according to the log files, no data was actually received, so there was no output. It showed a connection being made, which was immediately closed again.
The application has a little function that checks if the socket has been closed from the other side, by first calling poll() with a zero timeout, and if that says some data is available, it tries to read one byte. If that returns 0, the socket has been closed. But surely that must be a mistake, in this case. Is there a missing check for when errno is EAGAIN or something, an incorrectly handled timeout, or something? But no, no matter what I modified, the socket was definitely closed. The pair "you have data" plus "read returns 0" does indeed mean that the socket is closed.
The program used for sending the data, in this case, is netcat. It's part of most Linux distributions. Maybe it did something odd? Ok, let's run it under strace, to see if it actually sends anything. Running... and nothing. No output from it whatsoever. Huh? Oh. Everything here is running within Docker, and in this case in X86 mode on an ARM machine, so everything is emulated.
Then I found the netcat option "-v". This made it print something about "Preposterous fd value 18". What's wrong with 18? Everything above -1 is fine, stop complaining! Then I found a change from 2010 or something, increasing the max allowed fd number from 16 to 1024. This is still idiocy, but at least it would allow fd to be 18. The change was for version 1.10.30, and I was running 1.10.41. I checked which file descriptors were open before starting netcat, and now things started to make more sense. When the test was run as part of a suite, there was an extra level or two of shell scripts, and all of them required 2-3 processes as they all needed the rosetta emulator. I closed one of the descriptors under 16 before running netcat, and now the test passed every time. So, it seems Ubuntu had forgotten to include that change in their version. The BSD version doesn't have this check at all, but I didn't quite feel like trying to download and build that, all for a single test case with a functional workaround.
Netcat, in its infinite wisdom, had decided if the file descriptor value was "too high", it immediately closed it and gave up. No meaningful error message (unless run with "-v"), no nothing. Just "nah, don't wanna". File descriptors should be checked that they are above -1, that's all. They can, and will, be just about anything. Most likely relatively small, to make select() fast, but that's it. The newer poll() doesn't care how large the values are.
It may be a good idea to set FD_CLOEXEC on your own file descriptors, to avoid leaking them to child processes.
/Daniel