Kernel Call Trace trigger by rtpengine kernel module use

[rrb3942]

rtpengine version the issue has been seen with

Version: 12.5.1.35-1~bpo12+1

Used distribution and its version

No response

Linux kernel version used

6.13.12-zabbly+

CPU architecture issue was seen on (see uname -m)

x86_64

Expected behaviour you didn't see

Lack of a call trace.

Unexpected behaviour you saw

I am seeing a call trace being generate in the rtpengine kernel module, the trace is attached. The trace seems to generally be triggered around the first use of the kernel module by rtpengine. I haven't noticed it repeating, it seems to happen once and then not again (or very infrequently).

I have not noticed anything detrimental other than the logged trace, rtp seems to be forwarded fine.

KernelTrace.txt

Steps to reproduce the problem

No response

Additional program output to the terminal or logs illustrating the issue

Anything else?

In our setup, I consistently see it after a reboot and when rtpengine begins forwarding traffic.

[rfuchs]

The offending line is

          if (WARN_ON_ONCE(skb_shinfo(skb)->gso_type & SKB_GSO_FRAGLIST))
                  return NULL;

So apparently caused by hardware segmentation offloading.

What kind of network hardware/driver are you using?

As a workaround you can try disabling GSO on the network interface.

[rrb3942]

We are using Mellanox ConnectX-6 Dx virtual functions passed through to a vm.

I think the issue is that if gso is enabled the skb may contain multiple segments, and so they need to be split with skb_gso_segment and handled individually in that case?

[rfuchs]

I think the issue is that if gso is enabled the skb may contain multiple segments, and so they need to be split with skb_gso_segment and handled individually in that case?

Well, my understanding (I could be wrong) is that GSO is applicable to data to be sent, and not data that has been received, so I'm not sure why skbs with GSO would hit the netfilter rule. Perhaps the VM passthrough has something to do with it?

I tried reproducing this on a more usual setup but without luck.

Also generally GSO is used for TCP segmenting. Do you happen to have a custom network setup that might allow TCP packets to hit the netfilter rule?

[rrb3942]

rtpegine is managing the netfilter rule, so I don't think tcp is hitting it.

I'm not very familiar with the kernel net code, but after some digging this is what I found:

I think this caused by GRO support, which re-uses the kernel GSO flags and fields.

Look here:
https://github.com/torvalds/linux/blob/master/net/ipv4/udp_offload.c

In udp4_gro_complete() the gso_type has the SKB_GSO_FRAGLIST flag set. So if this path gets triggered there is the possibility that, on receive, you can get a skb with SKB_GSO_FRAGLIST set. The same behavior is present for the ipv6 version as well.

I believe these offload settings allow GRO for udp on supported nics:
rx-gro-list
rx-udp-gro-forwarding

This doesn't trigger when I try a simple test setup, so I do wonder if a single stream needs to have some unfortunate network jitter (or a sender is bursty) for rtp packets to arrive close enough to get coalesced by gro.

[rrb3942]

I think I have a partial working patch for this for ipv4 right now, will see if I can get it fully tested in the next week or so.

[rfuchs]

I think this caused by GRO support, which re-uses the kernel GSO flags and fields.

Ah good find, that makes a lot more sense.

This doesn't trigger when I try a simple test setup, so I do wonder if a single stream needs to have some unfortunate network jitter (or a sender is bursty) for rtp packets to arrive close enough to get coalesced by gro.

Or possibly by packets that are not RTP at all, but rather something else that happens to be in the same port range.

I think I have a partial working patch for this for ipv4 right now, will see if I can get it fully tested in the next week or so.

Perfect, let me know.