Problems accessing community.kde.org with non-1500 mtu connection

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Problems accessing community.kde.org with non-1500 mtu connection

Fabian Bläse
Hi there,

I'm having trouble accessing community.kde.org with internet connections with a smaller than 1500 MTU.
This affects IPv6 as well as IPv4.

It looks like ICMP(v6) Packet too big message are sent by the routers but are ignored by your servers.

pings with payload exceeding maximum packet size (1500 ping payload) on a 1500 mtu connections work (local fragmentation seems to be working).
However when trying the same on a 1492 MTU connection (or smaller), fragmented pings get sent, however I only receive the second (smaller) fragment. The first packet is too big to fit the connection, therefore a Packet too big icmp message is sent (which seems to be ignored).

I've tried various different connections (with routers from completely different nets issuing those icmp errors) with no success.

I've attached a tcpdump from a router sitting right behind the machine issuing those icmp errors.

I would appreciate if you could take a look into this.
Fabian

kde-icmp.pcap (12K) Download Attachment
signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Ben Cooksley
On Wed, May 1, 2019 at 7:38 AM Fabian Bläse <[hidden email]> wrote:
>
> Hi there,

Hi Fabian,

>
> I'm having trouble accessing community.kde.org with internet connections with a smaller than 1500 MTU.
> This affects IPv6 as well as IPv4.
>
> It looks like ICMP(v6) Packet too big message are sent by the routers but are ignored by your servers.
>
> pings with payload exceeding maximum packet size (1500 ping payload) on a 1500 mtu connections work (local fragmentation seems to be working).
> However when trying the same on a 1492 MTU connection (or smaller), fragmented pings get sent, however I only receive the second (smaller) fragment. The first packet is too big to fit the connection, therefore a Packet too big icmp message is sent (which seems to be ignored).
>
> I've tried various different connections (with routers from completely different nets issuing those icmp errors) with no success.
>
> I've attached a tcpdump from a router sitting right behind the machine issuing those icmp errors.
>
> I would appreciate if you could take a look into this.

I've investigated this issue, and it appears that our Web Application
Firewall service provider's systems block abnormally sized pings (with
IPv4 at least)
Therefore it would seem we'll need to take a different tack to
investigating this issue.

Would you mind describing the access issues you're having with
community.kde.org, and confirming whether this issue also affects
sites such as dot.kde.org, forum.kde.org and krita.org? If it does,
can you confirm that labplot.kde.org isn't affected?

> Fabian

Thanks,
Ben Cooksley
KDE Sysadmin
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Tobias Klaus
Hey Ben,

thanks for your investigation.

I am behind the same "low mtu" network and tested the sites you asked for.

Am Mittwoch, 1. Mai 2019, 09:48:35 CEST schrieb Ben Cooksley:
> Would you mind describing the access issues you're having with
> community.kde.org,
From a user's point of view the site https://community.kde.org suffers from a
timeout during the ssl handshake and is thus inaccessible.

> and confirming whether this issue also affects
> sites such as dot.kde.org, forum.kde.org and krita.org?
All theses sites time out the same way community.kde.org does.

> If it does,
> can you confirm that labplot.kde.org isn't affected?
As you suspected, labplot.kde.org loads just fine and is accessible.

Best regards,
Tobias

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Ben Cooksley
On Thu, May 2, 2019 at 9:32 AM Tobias Klaus <[hidden email]> wrote:
>
> Hey Ben,

Hi Tobias,

>
> thanks for your investigation.
>
> I am behind the same "low mtu" network and tested the sites you asked for.
>
> Am Mittwoch, 1. Mai 2019, 09:48:35 CEST schrieb Ben Cooksley:
> > Would you mind describing the access issues you're having with
> > community.kde.org,
> From a user's point of view the site https://community.kde.org suffers from a
> timeout during the ssl handshake and is thus inaccessible.
>
> > and confirming whether this issue also affects
> > sites such as dot.kde.org, forum.kde.org and krita.org?
> All theses sites time out the same way community.kde.org does.
>
> > If it does,
> > can you confirm that labplot.kde.org isn't affected?
> As you suspected, labplot.kde.org loads just fine and is accessible.

Okay. In order to escalate this to our provider we're probably going
to need some details on the characteristics of your connection.
Could you please provide the MTU of your connection, name of your
provider, along with the exact IPv4/IPv6 addresses which one of the
affected sites resolves to?

(Our provider has a global network, so your geographic location
changes the datacenter you get directed to)

Cheers,
Ben

>
> Best regards,
> Tobias
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Fabian Bläse
In reply to this post by Ben Cooksley
Hi Ben,

On 01.05.19 09:48, Ben Cooksley wrote:
> I've investigated this issue, and it appears that our Web Application
> Firewall service provider's systems block abnormally sized pings (with
> IPv4 at least)
> Therefore it would seem we'll need to take a different tack to
> investigating this issue
I've mainly tested IPv6.
The server does send correct replies to my big-sized IPv6 ping as can be seen in the previously attached tcpdump.
However they don't get fragmented correctly (according to the mtu sent inside the packet too big icmp) by your server.

> Would you mind describing the access issues you're having with
> community.kde.org, and confirming whether this issue also affects
> sites such as dot.kde.org, forum.kde.org and krita.org? If it does,
> can you confirm that labplot.kde.org isn't affected?
I've got the same problems as Tobias.

The network in question is 2a0b:f4c0::/32 (2a0b:f4c0::/40 in particular).
The MTU can vary due to different tunnel protocols used. The most commonly used MTUs are 1448 and 1420.
The ISP is F3 Netze e.V. (which is ourselves)

My own client is located inside 2a0b:f4c0:c8:6d::/64 however this problem seems to occur for the whole network.

I've also tested a different network (2a06:e881:3400::/44, icmp packet too big gets issued by 2a03:3f40:32::5165 there) and it seems to be broken there as well.

Fabian




signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Ben Cooksley
On Fri, May 3, 2019 at 12:06 AM Fabian Bläse <[hidden email]> wrote:
>
> Hi Ben,

Hi Fabian,

>
> On 01.05.19 09:48, Ben Cooksley wrote:
> > I've investigated this issue, and it appears that our Web Application
> > Firewall service provider's systems block abnormally sized pings (with
> > IPv4 at least)
> > Therefore it would seem we'll need to take a different tack to
> > investigating this issue
> I've mainly tested IPv6.
> The server does send correct replies to my big-sized IPv6 ping as can be seen in the previously attached tcpdump.
> However they don't get fragmented correctly (according to the mtu sent inside the packet too big icmp) by your server.

I see.

>
> > Would you mind describing the access issues you're having with
> > community.kde.org, and confirming whether this issue also affects
> > sites such as dot.kde.org, forum.kde.org and krita.org? If it does,
> > can you confirm that labplot.kde.org isn't affected?
> I've got the same problems as Tobias.
>
> The network in question is 2a0b:f4c0::/32 (2a0b:f4c0::/40 in particular).
> The MTU can vary due to different tunnel protocols used. The most commonly used MTUs are 1448 and 1420.
> The ISP is F3 Netze e.V. (which is ourselves)
>
> My own client is located inside 2a0b:f4c0:c8:6d::/64 however this problem seems to occur for the whole network.
>
> I've also tested a different network (2a06:e881:3400::/44, icmp packet too big gets issued by 2a03:3f40:32::5165 there) and it seems to be broken there as well.

Okay. This is a rather strange issue as we've not seen any other
reports of people having issues accessing our sites, which I would
expect to receive if people were having issues with IPv4. My own
ability to do local testing is limited unfortunately, but I can
confirm my connection's MTU for IPv4 is most definitely less than 1500
and has no issue accessing the affected sites.

Would you mind testing against our providers site, Imperva.com?

As you're an ISP it might be easier to put you in direct contact with
them as chances are their entire network is affected.

>
> Fabian
>
>
>

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Fabian Bläse
Hi,

On 04.05.19 09:52, Ben Cooksley wrote:
> Okay. This is a rather strange issue as we've not seen any other
> reports of people having issues accessing our sites, which I would
> expect to receive if people were having issues with IPv4. My own
> ability to do local testing is limited unfortunately, but I can
> confirm my connection's MTU for IPv4 is most definitely less than 1500
> and has no issue accessing the affected sites.
For IPv4 this might be hidden by the fact, that mss clamping (dirty hack, only works for TCP) is done pretty much everywhere with smaller-than 1500 mtu.
Even some really big companies like ubiquiti (who even make routers..) screw up the behaviour with icmp packet too big messages with IPv4 on their websites.

With IPv6 Path MTU Discovery gets far more important however, because it does not allow fragmentation on router level.

> Would you mind testing against our providers site, Imperva.com?
I can't verify the behaviour for that site, because it isn't even connected to the (non-legacy, ipv6) internet.. (doesn't have an AAAA record)
Jokes aside, it probably has the same issues.
But because, as you already noticed, pings with big payload get dropped, I can't investigate this further (because, due to broken PMTUD on many sites, we have mss clamping active for IPv4)

> As you're an ISP it might be easier to put you in direct contact with
> them as chances are their entire network is affected.
If you want I can contact them directly.

Fabian


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Problems accessing community.kde.org with non-1500 mtu connection

Ben Cooksley
On Sun, May 5, 2019 at 8:22 AM Fabian Bläse <[hidden email]> wrote:
>
> Hi,

Hi Fabian,

>
> On 04.05.19 09:52, Ben Cooksley wrote:
> > Okay. This is a rather strange issue as we've not seen any other
> > reports of people having issues accessing our sites, which I would
> > expect to receive if people were having issues with IPv4. My own
> > ability to do local testing is limited unfortunately, but I can
> > confirm my connection's MTU for IPv4 is most definitely less than 1500
> > and has no issue accessing the affected sites.
> For IPv4 this might be hidden by the fact, that mss clamping (dirty hack, only works for TCP) is done pretty much everywhere with smaller-than 1500 mtu.
> Even some really big companies like ubiquiti (who even make routers..) screw up the behaviour with icmp packet too big messages with IPv4 on their websites.
>
> With IPv6 Path MTU Discovery gets far more important however, because it does not allow fragmentation on router level.
>
> > Would you mind testing against our providers site, Imperva.com?
> I can't verify the behaviour for that site, because it isn't even connected to the (non-legacy, ipv6) internet.. (doesn't have an AAAA record)
> Jokes aside, it probably has the same issues.
> But because, as you already noticed, pings with big payload get dropped, I can't investigate this further (because, due to broken PMTUD on many sites, we have mss clamping active for IPv4)
>
> > As you're an ISP it might be easier to put you in direct contact with
> > them as chances are their entire network is affected.
> If you want I can contact them directly.

If you could contact them directly I think that probably would be the
fastest way to get a resolution on this (otherwise they'll respond to
me, then i'll forward it on to you, which will just delay things). If
you need details to contact them please let me know.

Sorry for taking so long to get back to you on this - things have been
quite busy lately.

>
> Fabian
>

Cheers,
Ben