RUN_ONCE and many flows fails

[bagder]

./client_http_run_once -n 80 bsd10.fh-muenster.de

and

./client_http_run_once -n 80 localhost

both show the same symptom (bsd10 is at a 42ms RTT from my test machine). It quickly handles the first 55-56 flows and then gets stuck on 25 or 24 flows remaining, spending 100% CPU time until after some time on_error is called and the application exits.

[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
[25 flows to go]
Waiting...
on_error

Running on Linux kernel 4.9.2, no working IPv6, no SCTP.

[weinrank]

Hey Daniel,
I'm facing the same problem and will try to solve it.

[weinrank]

I found the problem, it was a bug in my application, not related to the library.

[weinrank]

Hey Daniel,

https://github.com/NEAT-project/neat/blob/master/examples/client_http_run_once.c#L110

Here you call neat_stop_event_loop - why?
Is this useful if a ctx ist used by several flows?

[bagder]

I honestly can't remember any good reason for that. Possibly I added it during some debug session to see if it helped and forgot to remove it... !

[weinrank]

Should be fixed by #334

[bagder]

It works differently now, but it's still not working fine.

./client_http_run_once -n 80 bsd10.fh-muenster.de on my machine reaches further than before, but it gets stuck and never exits properly. The final lines of the output (before I pressed control-c) is below.

The cutoff line for me: at -n 65 it works, but at -n 66 it hangs. The last few lines in the hanging -n 66 case:

[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
Waiting...
[1 flows to go]
[Flow 29 ended, 0 to go, 0 for ctx 9]
on_close - no flows left for ctx, stopping event loop
[   0.890973][WRN] neat_notify_close - flow already closed - skipping
Cleanup!

Running it with gdb and breaking at that "Cleanup!" point, the stack trace looks like this:

^C
Program received signal SIGINT, Interrupt.
syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
38      ../sysdeps/unix/sysv/linux/x86_64/syscall.S: No such file or directory.
(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007ffff7970aea in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#2  0x00007ffff796ee34 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#3  0x00007ffff7960924 in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1
#4  0x00007ffff7b813ad in neat_close_loop (nc=0x6276e0) at /home/daniel/src/neat/neat_core.c:281
#5  neat_core_cleanup (nc=0x6276e0) at /home/daniel/src/neat/neat_core.c:290
#6  neat_free_ctx (nc=<optimized out>) at /home/daniel/src/neat/neat_core.c:329
#7  0x0000000000401779 in main (argc=<optimized out>, argv=<optimized out>)
    at /home/daniel/src/neat/examples/client_http_run_once.c:324
(gdb) 

[karlgrin]

@bagder @naeemk Not able to recreate the problem. Actually, it seems to work for me. Have anyone checked this issue recently?

[weinrank]

@karlgrin
I have and I still see the problem.
I've used the following command on a Ubuntu 17.10 machine.
./client_http_run_once -n 80 bsd10.fh-muenster.de

[karlgrin]

@weinrank @bagder
Have continued my investigation of issue #291. When I did a trial with a timeout of 10 ms for the poll() in client_http_run_once it seemed to work ok for very large values. For example, I ran './client_http_run_once -n 200 bsd10.fh-muenster.de' and it took outp.txt a bit less than 4 seconds to complete. You find the output enclosed with this email. My question is basically that I don't really see the problem.

[karlgrin]

@weinrank have you some time to start looking at this?