Zyxel T-56 ping spikes

  • 4 September 2023
  • 43 reacties
  • 1361 Bekeken
C
Reputatie 1

Hi all, pardon the engels, I am seeing the weirdest latency spikes to my new Zyxel T-56 modem, from devices directly attached to it. 

64 bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=0.555 ms
64 bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=0.533 ms
64 bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=516 ms
64 bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=0.578 ms
64 bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=210 ms
64 bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=0.568 ms

 

If I run a tcpping to google.com , so that is now LAN to WAN, and not just LAN to router. 

 

255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.677 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.382 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  554.650 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.165 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.706 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.652 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  6.536 ms
255  ams16s37-in-f14.1e100.net (172.217.23.206)  7.220 ms

 

I see the same thing?

This is weird how it gets these spikes, on LAN being directly attached I would expect to see max maybe 2ms. Not 516ms. Is this router getting overloaded, or is the hardware just kinda cheap? 

Tommie van Odido 8 maanden geleden

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

Bekijk origineel

43 reacties

Tommie van Odido
Rank
Reputatie 7
Badge +9

@ChaosMonkey, at the moment, I am not leaving or ruling anything out. The modem could possibly be a cause. I just want to hold out for another look from our Superusers! 

Life is too short to drink bad wine!
L
Rank
Reputatie 6
Badge

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

hi @ChaosMonkey : for Linux these test are not so dififcult  if you know how to use the command line.

Download the appropiate CLI client from https://www.speedtest.net/nl/apps/cli install it and run the test. I don’t know if they have a CLI client for non x86_64 hardware. When I run this client on my box I get something like:

[louis@lair ~]$ speedtest

   Speedtest by Ookla

      Server: ColoCenter bv - Zoetermeer (id: 3242)
         ISP: T-Mobile Thuis
Idle Latency:     4.72 ms   (jitter: 0.25ms, low: 4.46ms, high: 5.11ms)
    Download:   275.14 Mbps (data used: 311.8 MB)                                                   
                  4.93 ms   (jitter: 0.49ms, low: 3.92ms, high: 18.98ms)
      Upload:   389.91 Mbps (data used: 502.5 MB)                                                   
                  4.89 ms   (jitter: 7.02ms, low: 3.87ms, high: 222.81ms)
 Packet Loss:     0.6%
Re-running that test gives me better results (from another remote server)

I don’t feel however that this will give you much: your observation that there is a route flapping in your path indeed could be related. When the route (in the Odido network) flaps seems to be the point where packet loss occurs (the next hop after the first flapping router seems to show some packet loss when that happens. @Tommie van Odido: please ask the network guys to have a look at that. It may however be related to the second issue below when a DOS attack occurs?

The router going down when you are DOS’ed is another issue however.  The Zyxel should not be affected by it. You could try to disable the DOS protection under “beveiliging → firewall to see if that helps. It may be a case where the firewal is to pro-active and reporting on all threats (even on closed ports!) takes so much resources that the router hangs or reboots. @Tommie van Odido you may want to check with network people if this needs to be reported to Zyxel

The DOS attack may be triggered on the IP-address as soon as you get identified and you IP-address gets found out, so you cannot rule out that the attack is directed to you.

I may be wrong, but to me it all looks like being related to a DOS attack on you.

 

 

 

Ik ben ook maar een klant die wel eens andere klanten probeert te helpen.
C
Reputatie 1

Do you not have any latency spikes at all anymore after replacing the t-56? That sounds great. I think truly a firmware patch is needed and a look into what the cause might be to these spikes @Sven-Odido @Tommie van Odido 

Except for some oddness within the AMS-IX data center doing some odd route flip flop at a much higher rate than it should, my latency has been really stable. I used to feel that 200-400ms (600ms at times) latency spikes a lot in games as well. Since disconnecting the T56 I have had none of that. It’s been a month that it has been smooth. Except for that route change that will cause a spike, I don’t see the other constant or high 200-600ms spikes, within a 100 packets sent with a long running ping.

 

Curious enough, I can disconnect my Mikrotik and connect the T56, and within half an hour I will see the latency spikes. Disconnect the T56 and reconnect my Mikrotik, and boom immediate smooth ping. Same public IP address, or I can force a new one (in both swaps). So it’s not even that it’s an issue with my IP address. It seems to purely be the router. My guess is something is looking for these devices and then trying to do something towards them.

Tommie van Odido
Rank
Reputatie 7
Badge +9

Hi @ChaosMonkey, thanks for all the screenshots. I would like to ask if @TMTV and @louisL would like to assist you with the settings. I am also curious if they can see anything else in the screenshots with their expert eye.  

Life is too short to drink bad wine!
C
Reputatie 1

Hey @ChaosMonkey, hmm could you run a couple of speedtests for me? Then I will forward this to our tech guys.

The instructions for this can be found on our website. 

Please follow all the steps carefully, please share the screenshots in the topic instead of to our e-mail. We cannot process an application with missing steps.

Would this still be behind the Zyxel or behind my mikrotik as the edge router?

Curious why you have no guide for linux? That would be my go to, to make sure nothing is running the background.

C
Reputatie 1

Hi @ChaosMonkey, thanks for all the screenshots. I would like to ask if @TMTV and @louisL would like to assist you with the settings. I am also curious if they can see anything else in the screenshots with their expert eye.  

Hi, one thing that seems to stand out to me, is that the average response time of the modem seems to be an issue. Which could explain the odd avg’s seen in the pings to the google and cloudflare. If ICMP was being throttled that could explain it, but the network isn’t busy and a TCP ping rules that out. Seems the modem or the custom firmware T-Mobile (Odido) puts on this Zyxel might have an issue?

C
Reputatie 1

Hi @Tommie van Odido , 

 

Odido connects with smardc to the AMS-IX right? If I do a traceroute to 9.9.9.9 I see the same IP address twice. And it’s always when it goes from your 10.10.10.x range to the 80.249.x.x range where the latency spikes happen. (When the router isn’t being attacked and failing) .
When I do a traceroute to 8.8.8.8 (google) it doesn’t seem to run through the AMS-IX , and the ping is a lot more stable. Maybe there is an issue with your connection and routing at AMS-IX? 

 

Tommie van Odido
Rank
Reputatie 7
Badge +9

@ChaosMonkey, could you run the tests with only the Zyxel? 

We do not have a lot of customers using Linux haha. Therefore we do not have a Linus manual. 

Life is too short to drink bad wine!
Tommie van Odido
Rank
Reputatie 7
Badge +9

Hey @ChaosMonkey , this seems to be a possible denial of service attempt. More people seem to have reported the IP-address 185.191.225.130 as suspect. See for example https://www.abuseipdb.com/check/185.191.225.130 abuseipdb.comabuseipdb.com 185.191.225.130 | Probe Networks | AbuseIPDB 185.191.225.130 has been reported 142 times.

I am not sure how to tackle this issue. This may be an hacker attacking you (do you run a game where bad actors want to attack you?). But as you have a new modem (and a new IP-address) it may be directed to the previous user that had that IP-address. The destination port 8000 could also be the result of a service you have or had had open to the internet (forwarded to an internal server) that people want to disable (see  https://www.speedguide.net/port.php?port=8000 for some examples) SpeedGuideSpeedGuide

Port 8000 (tcp/udp) Port 8000 tcp/udp information, assignments, application use and known security risks. Was this the only message or did you receive more?

Life is too short to drink bad wine!
C
Reputatie 1

Seems it is based on the EX5601/EX5600-T SERIES 

eric
Rank
Reputatie 7
Badge +14

Hi @ChaosMonkey 

Did you use a wired connection between your test device and the router? Wifi is unreliable for such test.

Welkom bij de klant-helpt-klant Community van Odido.
C
Reputatie 1

Hi @ChaosMonkey 

Did you use a wired connection between your test device and the router? Wifi is unreliable for such test.

Yeah, you can see that from the first ping, the sub ms ping is normally not achievable on wifi. 
I am wired in with all my devices except my phone. 

Tommie van Odido
Rank
Reputatie 7
Badge +9

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

Life is too short to drink bad wine!
P
Reputatie 2

@ChaosMonkey 

 

Could you provide some screenshots (can use the ones here with the tests u did strictly with the zyxel). Maybe if we get more people to post it becomes a priority issue and we get a fix soon.

C
Reputatie 1

Hey @ChaosMonkey , this seems to be a possible denial of service attempt. More people seem to have reported the IP-address 185.191.225.130 as suspect. See for example https://www.abuseipdb.com/check/185.191.225.130 abuseipdb.comabuseipdb.com 185.191.225.130 | Probe Networks | AbuseIPDB 185.191.225.130 has been reported 142 times.

I am not sure how to tackle this issue. This may be an hacker attacking you (do you run a game where bad actors want to attack you?). But as you have a new modem (and a new IP-address) it may be directed to the previous user that had that IP-address. The destination port 8000 could also be the result of a service you have or had had open to the internet (forwarded to an internal server) that people want to disable (see  https://www.speedguide.net/port.php?port=8000 for some examples) SpeedGuideSpeedGuide

Port 8000 (tcp/udp) Port 8000 tcp/udp information, assignments, application use and known security risks. Was this the only message or did you receive more?

Hi @Tommie van Odido , so first off, I don’t have an ports mapped on port 8000. I was getting ping of death logs, and tons of these SYN flood attack logs, even with a new IP. So it’s definitely not targeted at me specifically. Odd thing is, this type of attack is normally not an issue on routers where it will cause a drop in the connection or a spike in CPU usage.


Now I cannot adjust the firewall rules in any meaningful manner, and that is what normally is done to stop this kind of thing. Which would also explain why this wasn’t an issue when I was using the mikrotik, since it would just drop this traffic and be done with it. 


As to why port 8000 shows up, that is something for Odido to address, since that isn’t something I configured nor can see that it is open. The games I play that might open a port with uPnP, but they don’t use port 8000, they either use a port in the 300-400 range, or in the 27k range. 

It is puzzling why it would lead to the device dropping the WAN connection.

C
Reputatie 1

For comparison, I used my own router, so just swapped out the Zyxel for a mikrotik hex (and it’s not even as powerful.) 

Fyi I connect to the Zyxel over a 2.5Gbps connection. 
 

Here is what it looks like using the mikrotik

 

 

So it looks like the custom firmware that is on the Zyxel from Odido(T-Mobile) could be the cause. 

  1.  Can I ask for a replacement router?
  2. Can I beta test new firmware?

 

C
Reputatie 1

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Hi Tommie, so I did some testing, seems the zyxel does indeed take a knock with the attacks. I can help beta firmware and see if that helps. 
For completeness sake, see the screenshot below. This was taken from my Mikrotik CCR2004 (there is no ways this connection will ever over load this device. 
 

What is a concern and what I would like to draw your attention to, is the fact that I am getting a route change between two IP’s at hope 2 and depending on the path hop 3 as well. now this shouldn’t be happening this frequently, this immediately make me wonder what exactly is going on?

C
Reputatie 1

Hey @ChaosMonkey, welcome to our Community!

Good that you are sounding the alarm for this. I immediately went to check for you and I see that you have your own router/switch connected. Could you run these two commands if you have unplugged everything and are only connected directly wired to your laptop/desktop? Ping -n 50 1.1.1.1 & Ping -n 50 8.8.8.8 in CMD. Do this with only the Zyxel modem connected to your network with a wired connection. Suppose you have a VPN, disable it too.

What do you mean you can see I have my own router/switch connected? The tests I showed was done from a linux machine directly connected to the Zyxel router. Even with just my PC connected it still shows odd spikes, and I was initially talking about ping spikes to the Zyxel router itself. That is why my concern is sitting 
 

 

S
Rank
Reputatie 7
Badge +4

@Sven-Odido you can have a look now, maybe we need a faster way to communicate, because these attacks come the moment this router turns on

I havent had time yet to dive into this. When a lot of port scans etc are happening at the same time its normal to see a slight increase in latency for a few pings. But not the amount you are seeing. I will have to look more into that when I have more time on my hands, Sorry for the inconvenience.

Also keep in mind that when a lot of ICMP pings are sent by one host, the DOS protection on the T56 will kick in as well. 

Helpt hier soms Tommie van Odido
C
Reputatie 1

@Sven-Odido you can have a look now, maybe we need a faster way to communicate, because these attacks come the moment this router turns on

I havent had time yet to dive into this. When a lot of port scans etc are happening at the same time its normal to see a slight increase in latency for a few pings. But not the amount you are seeing. I will have to look more into that when I have more time on my hands, Sorry for the inconvenience.

Also keep in mind that when a lot of ICMP pings are sent by one host, the DOS protection on the T56 will kick in as well. 

Not a problem, would it then be okay if I swap back to the mikrotik and only switch to the T56 when you have some time? 

 

I am curious ipv6 needs ICMP in order to work, is the T56 DoS protection setup in a way that it would allow that without raising any flags? It seems to only trigger on the ICMP request count number, so if you ping and stop and ping and stop it seems to not flag it. 

P
Reputatie 2

Nothing from my pc to the router, except a 5ms spike that I only saw once, rest of them were all sub 1ms. The issues seem to be strictly when connecting to the actual internet, not LAN. And no idea about the first part to be honest, I just picked up that it flags my computer sending ping of death attacks to multiple gameservers, doesnt seem to be anything else though like google servers or twitch etc...

S
Rank
Reputatie 7
Badge +4

Not a problem, would it then be okay if I swap back to the mikrotik and only switch to the T56 when you have some time? 
​​​

Yess Of course, I will let you know when the T56 need to be connected again. 

 

Helpt hier soms Tommie van Odido
C
Reputatie 1

Hey @ChaosMonkey, that's an interesting comparison. I have sent you a new modem just to be sure. Could you test this one out and provide me with feedback?

Will do. 

 

Also there seems to be something unstable happening in the routing. 
 

hop 2 and 3 show route changes, so I am getting unstable latencies 

Tommie van Odido
Rank
Reputatie 7
Badge +9

Hey @ChaosMonkey, can you take a screenshot of the results instead of the hops? Also, I would like to ask you to make a sketch of your home network. What we suggest is ONT/Media Converter -> Zyxel -> Wired connection laptop/other device. I am very curious to know what else is in between.

Life is too short to drink bad wine!
P
Reputatie 2

@ChaosMonkey Did you ever end up resolving this issue? Me and some others are experienceing ur same issue.

Reageer


ForumvoorwaardenCookie instellingen