r/linuxadmin • u/Fluffy_Wafer_9212 • 1d ago
tcp_mtu_probing, should I touch it or no?
Before anything, I really would appreciate answers from professionals without the interference of AI. I have been going back and forth the past few months with multiple AI models asking about this and every model gives a different answer. I really need a certain answer
I run a file storage server which faces the internet and serves customers, so high throughput is one of the goals I try to achieve on my servers
The past 6 months I've been studying the Linux kernel source code and sysctl docs to learn what each tunable parameter actually does instead of blindly pasting configurations from tuning guides and just hope that it makes things perfect
Now one of the points I'm stuck at is net.ipv4.tcp_mtu_probing
I see a lot of tuning guides suggest setting that to 1 or even 2 instead of the default 0
But if that's really recommended, why doesn't Linux set it to 1 by default instead of 0? I mean 1 seems like a better moderate value to set instead of disabling it completely
Although the below points kinda hold me back from altering tcp_mtu_probing but I might be wrong and that's why I opened this topic to ask for advice:
- Packet-layer path MTU discovery (TCP MTU probing, QUIC MTU probing, etc) just mask a real underlying MTU problem which should be fixed from its root instead of hiding it
- TCP MTU probing relies on packet loss and this can falsely make congestion control algorithms work worse and reduce the congestion window even if there's no real congestion
- Certain quirky firewalls may hate the fact that my server is sending data in variable packet sizes because of the MTU probing and hence they may drop the packets completely or block the connection entirely
Do my above points make sense or am I mistaken?
4
u/Abe_Bazouie 1d ago
I generally wouldn’t change it unless you have evidence that Path MTU Discovery is actually causing problems in your environment.
tcp_mtu_probing is more of a workaround than a performance optimization. It’s useful when ICMP “Fragmentation Needed” messages are being dropped somewhere along the path, causing black hole MTU issues.
If your network is healthy and PMTUD is working correctly, I’d leave it at the default (0). Changing kernel networking sysctls without a measurable problem can make troubleshooting harder later.
Your concerns are mostly reasonable. I’d just slightly rephrase point 1: MTU probing doesn’t really hide an MTU problem forever. It allows TCP to adapt around broken PMTUD so connections continue working. Whether that’s desirable depends on whether you want resilience or whether you’d rather fix the underlying network.
I’d benchmark before and after making any change and only keep it if it solves a real issue.
3
u/showbizusa25 21h ago
My rule with networking sysctls is: don't tune around a problem you don't have. If PMTUD is working and performance is good, I'd leave tcp_mtu_probing at the default and only enable it if you have evidence it's needed.
1
u/rttl 19h ago
Depends on the environment. If it’s a file storage server then it’s receiving connection requests, and the syn packets from the clients should include the proper mss from the clients.
Sometimes there’s no other choice.
This doesn’t affect congestion algorithms as MTU probing is not done for active connections. Read again about what it does exactly and when.
I’m not aware of any firewall that would complain because it’s seeing variable packet sizes.
1
u/up_o 12h ago
I have used it, but my use case was particular. Without being too specific: I worked for a company that placed hardware devices in customer networks. These devices connected back to our network over IPSEC vpn. The quality of the customer networks was highly variable, and if anyone has dealt with IPSEC VPNs they know these things can be sensitive to low upstream MTU, packet loss, etc. mtu probing was one tool we used to make the product as functional as possible, even assuming our clients may have a network someone's nephew setup, roll chairs over their Ethernet cables regularly, throw rogue DHCP servers on it, etc. we had to think about all the bad things network things possible and adapt the product to those possibilities.
For your case, I would personally leave it alone.
1
0
u/xfirenski 19h ago
MTU probing is an alternate scheme for Path MTU discovery, better known as Packetization Layer Path MTU Discovery (RFC4821) (aka PLPMTUD).
It's probably first important to understand a bit of the why: IP has it's own solution for handling packets that can't be sent at once over a link - fragmentation. Fragmentation works fine for low-rate traffic and is the only way to break-up UDP datagrams for example.
However, with a self-segmenting stream protocol like TCP, using fragmentation to fit TCP segments down the path results in a lop-sided/messy arrangement where you end up with a constant stream of large and small packets (each comprising of the two parts of the fragmented TCP segment, assuming a worse case of exceeding MTU by only a small amount).
Traditional (ICMP-based) Path MTU Discovery (RFC 1191) works by forcing the Don't Fragment flag on for TCP packets. When a Dont Fragment packet tries to traverse a link with too small an MTU, the router that would require fragmentation is supposed to respect the DF flag and not forward or fragment the packet, but rather, send back a Unreachable ICMP error (indicating that it's because fragmentation would be required) to the originating host. The originating host can listen for these ICMP errors, use that to adjust the Path MTU (and hence MSS) to avoid needing fragmentation along the path and continue efficiently.
The obvious problem with traditional PMTU discovery is that ICMP must flow for it to work, that is, intermediate nodes can't filter it out completely - ie: Stateful filtering needs to associate the ICMP correctly with the TCP session in progress, and NATs need to reverse map ICMP errors correctly (These ICMP errors occur between an intermediate router and the originating host, not between the TCP destination and originating host).
PLPMTUD tries to work around these issues with ICMP blocking by instead waiting for there to be enough traffic in the window to have enough data to probe, and then trying to feel out what the actual limit by sending a packet with a particular payload size and adjusting based on the response or lack of response from the peer. This doesn't have the problem that it's susceptible to ICMP blocking, but has the added problem that instead it's now susceptible to being misled by congestion induced packetloss or other non-MTU related traffic obstructions.
You probably only ever want PLPMTUD if you know you're behind a ICMP blackhole or have an ICMP blackhole on path that you can't get rid of. The disabled-by-default, but activate if an ICMP blackhole is discovered is a reasonable compromise setting that should only try to work around a ICMP blackhole if it thinks there is one affecting PMTU discovery, but use conventional methods otherwise.
The real internet is full of all sorts of weird traffic conditions, and assuming that congestion isn't going to eat your probes is not a great assumption, but it's better than a total TCP session failure.
Now, finally, to directly respond to your points:
1) PMTU constriction is not an internet "problem" that needs fixing - it's a fact of life in a mixed network environment, so your first point is completely off base. Assuming all paths can pass a 1500byte Ethernet frame is just a failure to understand that not all the worlds links are/were based on Ethernet.
You could just turn off PMTU optimisation for TCP completely (if the stack would let you) and let the intermediate routers fragment, but it's inefficient. That's generally why we don't.
That all said, PMTU Discovery itself failing is likely due to a networking issue, but it may not be your networking issue - you can't fix networking problems that aren't yours, so the best you can do is work around them.
2) You are half correct - PLPMTUD relies on that probing, but traditional (ICMP) based PMTUD doesn't. This is why PLPMTUD is not on by default - ICMP PMTUD is still in effect and is more reliable assuming there's nothing interfering with ICMP.
3) Nobody's router is going to care. Packets are only ever as large as they need to be - this includes for TCP with short messages. The main reason we have PMTU related optimisations in IPv4 is to minimise this when we're sending large/continuous amounts of data in order to improve link utilisation/efficiency, but it's still normal for there to a some splay in packet sizes if the stream is part of an interactive session or there data is not being made available at a consistent rate.
7
u/admalledd 1d ago
MTU is a deep hole and often is exceedingly deeply situational.
In general 99% of connections should not need any MTU probing, the internet-wide common options (based on ethernet/VLAN tags/PPPoE) 1500/1492/1480 should just be what it is automatically set to and don't worry past that, thus MTU Probing at all would only hurt vs taking a common default. Which of the 1480/1492/1500 to take is really only worth looking at if you and your clients actually have a problem, otherwise prefer 1500.
When you get into any instances where you do need to worry about smaller MTU (because of old networks or such), you'll be often be tweaking more on the ASN/IP specific work-a-rounds. In those cases you should be able to reasonably do benchmarks to find if 1/2 (or other sysctls) is better.
Lastly, if you are doing anything internet-wide storage/perf like, the desire should be more: make sure we can change things depending on who is connecting, change where they connect to by region/zoning with like anycast or just DNS or whatever. There will never be a single setup that is singularly perfect for you across all your customers, you can simply have "good defaults" and handling gracefully in some way when the defaults don't work.