Instant On - Wireless

Expand all | Collapse all

Problem AP22 and 2.3.1

  • 1.  Problem AP22 and 2.3.1

    Posted 07-12-2021 04:41 PM
    Having major problems with 2.3.1.  Macbooks seem to have an issue where they are connected just fine, but can't contact their gateway on occasion.  If I get 2 pings going, one to a local server, and one out to the internet, the one to the local server doesn't drop. However the one out to the internet will drop for maybe 20-30 times in a row and the magically come back. 

    These are 3 AP22s connects to a 1930 switch, with a SonicWall as a gateway. I have confirmed the SonicWall isn't blocking anything.  The same Macbooks I can connect to the ISP provided wifi with 0 issues. So it's nothing like that. 

    I don't know, these AP22s have been solid, and now they are just flaking out everywhere even though they are up to date and I've rebooted them.  I'm always at max signal on the macbooks. Real worlds transfer speeds are above 300mbps. They just suddenly can't communicate with the gateway, yet an iphone right next to it, connected to the same SSID, is sitting there running speed tests without any drops, all while the macbook can't get to the gateway.

    I've turned off Wifi6.  Not sure what else to check.


  • 2.  RE: Problem AP22 and 2.3.1

    Posted 07-13-2021 10:24 AM
    Hi,

    can you confirm that turning off WiFi6 knob helped resolving the issue?
    Also, if you can confirm your MacOS version and Macbook type that'll be appreciated.

    Thanks,

    ------------------------------
    Jul.
    ------------------------------



  • 3.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 10:48 AM
    Hi,

    We want to assist you in getting this issue resolved. We request you create a support ticket with our Aruba Instant On support team and include setup, all client issues seen, and any additional concerns relating to the issue. This will help our team assist you in narrowing down and resolving the issue. You can reach us at: https://www.arubainstanton.com/contact-support/


    ------------------------------
    Aruba Instant On Communications
    ------------------------------



  • 4.  RE: Problem AP22 and 2.3.1

    Posted 07-13-2021 10:36 AM
    2018 Macbook Pro.  Fully up to date Big Sur.  Turning off WiFi6 had no effect.

    I'm still testing, but so far it seems to only happen on an SSID that has both 2.4 and 5Ghz enabled. Last night I tested on an SSID with only 5Ghz, with WiFi6 ON and optimize for video streaming ON and it didn't drop at all. I'm back on a combined SSID now to verify issue returns.  If it does, next I will try 2.4Ghz only to see how that is.

    Thanks,

    Rob


  • 5.  RE: Problem AP22 and 2.3.1

    Posted 07-13-2021 01:10 PM
    I can now confirm that the problem exists on SSIDs with both 2.4Ghz and 5Ghz enabled. Leaving all other settings the same, simply removing 2.4Ghz from the SSID resolves the issue.  Soon I will test with 2.4Ghz only to see if that has the same issue as the combined.


  • 6.  RE: Problem AP22 and 2.3.1

    Posted 08-09-2021 09:35 AM
    Just adding this is still going on. I even added a 4th AP22 to rule out a hardware problem and while connected to it the problem happens. The problem doesn't happen while connected to AT&Ts provided Harris CPE. 

    It also happens on all bands, it's not isolated to 2.4Ghz. Even on a 5Ghz only SSID, with Wifi6 and media optimizations disabled, it still happens, just not as frequently.  It's obvious streaming something like HBOMax which tends not to buffer as much. You'll be watching and then it suddenly stops and says can't connect. I look at my pings and sure enough, pings to a local server are fine, but pings out the gateway drop. I can't even ping the local gateway (SonicWall) itself during this so it's not an internet problem. This persists for perhaps 15 seconds then resumes and is fine again for a couple hours usually before it happens again. idk, something is definitely wrong. It doesn't happen on wired connections. It doesn't happen connecting with WiFi to the AT&T CPE. It only happens while connected to AP22s.


  • 7.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 11:39 AM

    Interesting (and frustrating) problem! 

    You initially said this was only affecting MacBooks, while iPhones were okay. Does that still hold true, or have you discovered otherwise? 



    ------------------------------
    Ethan
    ------------------------------



  • 8.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:03 PM
    The problem is that it's hard to test. It can go several hours with no issue then it hits. The MacBook is used heavily for streaming video so it's obvious when it happens. None of the other devices are used in such a way where they could easily detect a drop. When the drop happens, it's usually for about 45 seconds. Enough to stop streams.  I haven't noticed it on other devices, but I can't rule it out either.


  • 9.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:11 PM

    Ugh. These types of issues are tough to diagnose if you can't be onsite and don't have remote monitoring. 

    We've been seeing similar issues with our provider here in Western Canada, who uses the same equipment as Comcast. It's their modem/routers (XB6&7) that are having the issues. 

    Have you double checked to be sure there's no accidental QOS / Application blocking or similar settings in the router and APs that might be choking the video feeds? Don’t forget that with an errant click you can neuter application & traffic types.

    Again - is it only Video streams that are affected? 

    Are you familiar with Wireshark? It would be interesting to fire it up when there’s an issues to see what the frames and packets are doing. 

    ------------------------------
    Ethan
    ------------------------------



  • 10.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:22 PM
    Yes I'm familiar with WireShark, it's hard to get the timing though. I would have to sit and sit waiting.  There's no QoS or bandwidth mgmt in use. It's a synchronous 1Gbps circuit. I leave 3 pings running on the MacBook.

    1 to local gateway (sonicwall)
    1 to 8.8.8.8
    1 to a local server

    During the event. The last one, to a local server in the same subnet, that never drops. It's always fine. During the event, all pings to the local gateway, and then obviously out to the internet from there, fail. Even with QoS or BWM, it still wouldn't drop all pings for 45 seconds to the local gateway.


  • 11.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:29 PM

    All three pings are being transmitted over wifi?
    If so, I don’t think you have a wifi issue, you have a routing issue. Do you have another router you can drop in place of the sonic wall?

    ------------------------------
    Ethan
    ------------------------------



  • 12.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 06:39 PM
    Yes all 3 pings are being transmitted over wifi. The macbook is not wired up, it's only wifi. However, I disagree with the routing problem assessment. The SonicWall I had was a TZ400 and I had the problem. I recently replaced it with a newer TZ470 and still have the problem. There are no other routers.  All 4 APs connect to an Aruba 1930 switch which uplinks to the SonicWall. All other wired devices on the same 1930 switch have no issues, ever. It's a flat network, no subnets or VLANs. It's not the Macbook itself, as the problem never, not even once happens, while I'm connected to directly to the ISP CPEs wifi. It only happens while connected to these Aruba AP22s. It also never happened before. One of the recent updates is when it triggered.


  • 13.  RE: Problem AP22 and 2.3.1

    Posted 08-13-2021 02:17 AM

    You have me worried now. I'm going to pull an AP22 off the shelf and start testing. Better that we discover an issue in the lab instead of a client's in-production network. 

    If I can replicate the issue I'll report back. 

    Just thinking: The ping to your local server is over layer 2. It never drops, right? It's only the layer 3 pings & traffic that are dying. Doesn't that squarely point back to the router? WiFi only exists in layers 1&2. If the wifi was the problem, why aren’t your pings to the local server also dying? They’re literally running over the same layer 2 network.

    We know traffic is flowing most of the time - so what could be intermittently disrupting / corrupting traffic? 



    ------------------------------
    Ethan
    ------------------------------



  • 14.  RE: Problem AP22 and 2.3.1

    Posted 08-13-2021 08:00 AM
    The gateway itself is also responding at layer 2. Pings to the gateway fail. Now the ping out to 8.8.8.8 would be operating at layer 3. 

    Client IP:  192.168.1.50
    Gateway: 192.168.1.254
    Local server: 192.168.1.10

    Pings from that client, to either the gateway or local server operate at layer 2 only.  There is no routing in this scenario, it's all layer 2.  The only time layer 3 routing is involved is when I'm pinging 8.8.8.8.

    So if a ping fails to 192.168.1.254, and not to 192.168.1.10, that has nothing to do with routing, and everything to do with the APs.  That's why I disagreed with the routing issue assessment.

    I'll add that by this point, I'm using a new SonicWall and a new AP22 in the location. So neither of those are the culprit in terms of possible faulty hardware. The only other item that hasn't been replaced is the 1930 switch, but wired clients on it never have an issue, so I don't believe it is the culprit.

    What's interrupting traffic is a bug with the APs.


  • 15.  RE: Problem AP22 and 2.3.1

    Posted 08-22-2021 05:51 AM
    If ping to local server 192.168.1.10, then I think it is not wifi issue. If you have same result when ping to gateway and local server, then it will be wifi issue

    ------------------------------
    DENNY .
    ------------------------------



  • 16.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:00 PM
    I am sorry but that's just not true. I have witnessed similar bugs on other deployments with different gear. The local gateway is the same thing as a local server. So I completely disgree with that assessment.

    I also have more notes now.

    1. The issue affects all devices. Not just Macbooks. It I have verified that Apple TVs, Macbook Pros, Macbook Airs, and a Windows Laptop all have the issue.
    2. It's not DHCP Renewal related. The issues happens even when statically assigned.
    3. I've maually channelized the network, dropping 40Mhz wide channels instead of 80, and avoiding all DFS channels. The issue still happens.

    So I'm at a loss now. This wasn't there with previous firmware.  

    Pings are consistent as well.  After running overnight:

    Pings to local server 0.5% loss
    Pings to Gateway 1% loss
    Pings to Internet 1.5% loss.

    So the loss is only 1% to the gateway but what sucks is almost all of that loss is in a chunk of 60 dropped packets in a row which breaks connections, streams, VPNs.  This happens I would say on average about once per hour to all clients connected to these 4 AP22s.  None of this is witnessed when just connecting to the ISPs Harris device.

    New firewall, new APs, none of that solved it.  All other wired machines on the same 1930 switch can ping the gateway all night long without a single dropped packet. So it isn't the wired network either. All 4 AP22s are home run to the 1930. No mesh is in use. One of these recent firmware updates triggered this.


  • 17.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:08 PM
    Oh and when it happens, it happens simultaneously to all clients. At least all clients on the same AP.  I need to do some more testing simultaneously from multiple APs to see if it syncs up with all APs, or if it happens individually per AP, affecting only those clients on that AP.  I know for fact that at a minimum, when it happens to 1 AP, it affects all clients at the same time, in the same way, who are associated with that AP.


  • 18.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:14 PM

    Thanks for the additional details. 

    We've been on vacation and unable to test this is the lab, but it's going to be one of the first priorities once we return. 


    I’m assuming no one from Aruba responds to these forums? Have you opened a ticket with support? 



    ------------------------------
    Ethan
    ------------------------------



  • 19.  RE: Problem AP22 and 2.3.1

    Posted 08-29-2021 04:27 PM
    What SonicWALL firmware are you using?

    ------------------------------
    Stephan Gladiadis
    ------------------------------



  • 20.  RE: Problem AP22 and 2.3.1

    Posted 08-30-2021 03:26 PM
    SonicOS 7.0.0-R906 on a TZ470.  But the problem started before I swapped out the older TZ400 that had different firmware. So I don't think it's SonicWall related at all for several reasons.

    1. The SonicWall was upgraded with a new device and new firmware, didn't fix this issue.
    2. There is never an issue on the wired network. You can ping the SonicWall at 0% loss all day from any wired endpoint including endpoints on the same 1930 switch as the AP22s.
    3. It's a flat network with a single subnet and no VLANs.
    4. There are never any issues listed in the SonicWall logs that could even remotely be related to this.



  • 21.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 04:45 PM
    Hi,

    I am dealing with a similar behavior here. I have 3x AP22 running, with an Aruba 1930 and a Ubiquiti EdgeRouter 4.... I am using my MacBook Air 2020 to stream using AirPlay2 over an Apple TV 4 HD. No matter what I do, it is always have dropping at an intermittent amount of time... 5 min, 1 hour.... no specific time.

    This is really frustrating. I am using this Airplay2 streaming as my 2nd monitor, and the streaming is dropping quite many times per day. I carefully followed Apple advices on how to choose right WiFi settings: channels, bandwidth, etc.

    Not much luck so far.

    I re-tested with my old all-in one wifi-router from Synology RT-2600AC alone... nothing else.... everything is stable, no issue at all. The reason I changed everything here is to let me have a better WiFi coverage.

    I am seriously considering dropping all my Aruba gears for another vendor.

    Jeff

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------



  • 22.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 05:17 PM
    That sounds identical to what I am dealing with. It wasn't always like this though, a recent firmware update caused it.  I am currently on the phone with Aruba creating a case. For me, the lengthy drops could be anywhere from 15 to 60 seconds long, and can happen anywhere from 1-4 times per hour.  I haven't been able to find a pattern to it. But I know it's not specific to the AP as I have 4 of them and they all have the issue.

    I wouldn't drop them yet. I think this is just a bug for now. I came from Ubiquiti to these and these were night and day better than Ubiquiti a first, but now the firmware updates broke them a bit.


  • 23.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 05:21 PM
    I'm glad we have corroboration on the issue! It sounds like the team at Aruba are listening, so let's hope they flesh out the bug and get things back on track!

    ------------------------------
    Ethan
    ------------------------------



  • 24.  RE: Problem AP22 and 2.3.1

    Posted 09-16-2021 02:18 PM
    Just as an update, I opened a case a couple weeks ago. I have been testing the living hell out of this.  Aruba support felt it might be the 1930 switch causing an issue, so they moved it to local config mode, and made changes to LLDP. However, that didn't fix anything. They have reviewed everything and can't find an issue. I just got an update that they are escalating this to engineering.  

    So we aren't sure yet if it's the 1930 switch, or the APs.  All APs are connected directly to this 1930 switch. Wired clients on the 1930 switch have no issues ever and can ping the local gateway all night long and not drop a single packet. Wireless clients connected to AP22s, that are subsequently connected to the 1930 switch have this issue where they may drop as many as 80 pings in a row to the local gateway, often several times per hour, all while also still being able to ping a local server. It's really bizarre how they are denied access to only the local gateway, and subsequently the internet, but not other hosts. Again, wired clients on the 1930 switch never ever drop to the local gateway.

    Does anyone else have AP22s connected to a 1930 switch? If so, could you do some extended pings to your local firewall and see if the issue can be reproduced?  I think I might take one of my AP22s, and connect it separately to a different switch to see if it's the 1930 causing it, or the APs.


  • 25.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 06:48 PM
    So far, my work around is not very gracious..... I brought back to life my old Apple AirPort Extreme wifi-router. I created a new and exclusive 5 Ghz wifi SSID... 2.4 Ghz disabled. The 5G network is running only on 1 band that I now use with my Apple TV 4 HD exclusively... for now. I also disabled the band I am using on Aruba network to avoid conflict.

    I left my MacBook Air on Aruba WiFi network to test this out. If things are not getting better, my next step would be to move the MacBook Air also on the same 5G network SSID as the Apple TV 4 HD. Problem still persist.  UPDATED

    UPDATED : Both MacBook Air and Apple TV 4 HD are now on the same 5G network using Apple AirPort Extreme router to avoid dropout.

    It was working with my old Synology too.... But I want to try also with older gears too. 

    It is indeed IMHO related to the use of Aruba AP. 

    Regards.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------



  • 26.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 10:15 AM
    Hi,

    Could someone corroborate this behavior?

    I have 1 SSID only using both 2.4 and 5 bands. The global radio settings is set to use power, for both bands, from low to high... not very high or max..... not more than HIGH period.

    From Aruba portal, I can also confirm that I did not override any radio settings for all 3 AP22.... the override checkbox for specific settings for each AP is not checked for all AP22.

    But if I take a look at each AP22 status to check which power level they are dynamically using.... I have one AP that power is set momentarily to MAXIMUM and another one to VERY HIGH.

    So my understanding is even with global radio settings threshold is set to use power level between LOW and HIGH not much, it does not seems to work at all ????

    Am I the only one?

    Regards.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------



  • 27.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 11:32 AM

    I have only experimented with a few APs' power settings, but found we had very inconsistent performance when the power was set to variable or anything less than High. Not just a small dip in performance, but a drastic reduction in throughput. 

    Have you opened a ticket w/Aruba, Jeff? 



    ------------------------------
    Ethan
    ------------------------------



  • 28.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 12:03 PM
    Hi Ethan

    No, I did not open a case yet. I brought this issue to their attention back in July and received a few tweaks to make it work, such as using less 5G bands, etc... but that do not solved my problem. At that time, I gave up unless I found something relevant to tell them. Going forward, if they do not provide me real explanation.... I am not loosing my time.

    Like I said, I have 3 AP22.... They are all running at AVERAGE power level. But one of them is running at VERY HIGH power on 2.4 and the other one is running at MAXIMUM power on 5G band only. So far, signal/speed level are all green (good) if I refer to the dashboard. I understand that they manage their dynamic power output properly and I have so far no other issue as well.

    But my concern is "Why VERY HIGH power and MAXIMUM power are set when I asked to not go beyond HIGH only ???

    Regards,

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------



  • 29.  RE: Problem AP22 and 2.3.1

    Posted 09-11-2021 09:45 PM
    What are their dBm levels?  When I change mine the names may still say very high or max but their stated dBm value is lower.  Also there are two places to change those setting which is odd.  On the individual AP page on the bottom under radio management and then again on the inventory page under the three dot menu theirs another radio management which you would thinks changes all the APs whereas on the actual individual AP page you would think that would change just that AP but the one on the inventory page doesn't seem to change the individual pages.


  • 30.  RE: Problem AP22 and 2.3.1

    Posted 09-12-2021 07:50 PM
    Hi,

    I can't tell. Since my last post, I did a wifi site survey to determine a different channel for each AP on both 2.4 and 5. Then I manually setup each AP with their own channel and power level. Power level is now set MIN = MAX... same for both. So power level is always the same now. I do not let AP manage channel and power now by themselves; to many times I saw 2 AP running on the same channel.

    I still can't stream using Airplay from my MacBook Air 2020 to my Apple TV 4 HD without having disruption over time. It is frustrating..... but I cannot be sure at 100% it is related to a WiFi issue.... Airplay is always using WiFi, even if the Apple TV is wired. I will wait for a next major Apple Mac OS release soon to check if the issue still occur.

    Unless my Airplay issue, my network is very efficient and stable.

    Regard.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------



  • 31.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 10:37 AM
    Sorry no useful input, but just to say thanks for highlighting this.

    I too am looking to jump out of the Ubiquiti ship, due to continuing instability, and was going to replace all switches and APs with Instant On. I had heard that Aruba has a good reputation for stability, which is what I need above all else, but now I'm not so sure.

    Waiting very interestedly to see the outcome of this issue before making any final decision.

    ------------------------------
    Keith Richards
    ------------------------------



  • 32.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 06:03 AM
    Progress! When removing an AP22 from the 1930 switch, and connecting it instead to the upstream switch, the problem is resolved for anyone connecting to that AP. I just went 12 hours without the problem happening once whereas before, it would happen several times per hour.
    So the problem isn't isolated to the AP22s, it's somehow a combo problem when an AP22 is directly attached to the 1930 switch. Other wired clients on the 1930 switch still never have an issue. It's only when connecting via AP22, and only if that AP22 is connected to this 1930 switch. So the 1930 switch works perfectly, for anything other than an AP22. The AP22 works perfectly, as long as it isn't attached to the 1930. But put an AP22 on a 1930 and things break.
    Today I am going to try the reverse, and put a PoE unmanaged switch downstream of the 1930 switch and connect the APs to it, with the unmanaged switch uplinked to the 1930. 
    But it does appear to be some type of negotiation issue between AP22s and the 1930 switch.



  • 33.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 06:17 AM
    Jeff see my post above. Can you try moving an AP22 off of the 1930 switch in some manner and see if the problem is resolved for guests on that AP? Just hook it up to some other PoE switch, or any switch and use a PoE adapter. My problems disappeared when I moved the AP22 to a different switch. Would like to see how this affects yours as well since I am sure you have the same issue, as you also have a 1930 switch.


  • 34.  RE: Problem AP22 and 2.3.1

    Posted 24 days ago

    Has Aruba actually acknowledge this issue yet?  Usually a company with let the community know they are aware of the issue and if they were able to replicate on their test bench.  At least then we would know there are engineers actually working on this issue.  Their silence isn't very assuring that a fix might be in the works and its not very assuring that this product line is getting the attention it deserves or we deserve.  Reminds me a lot like Cisco and their SMB line and now long it took to get critical updates for basic functions to work properly.  

    If I have to pull these switches in order for the AP22 wifi networks to be reliable I will but I won't be using these switches in the future if Aruba can't even keep us informed or even acknowledge the issue and that's its been replicated.