Instant On - Wireless

 View Only
Back to discussions
Expand all | Collapse all

Problem AP22 and 2.3.1

  • 1.  Problem AP22 and 2.3.1

    Posted 07-12-2021 04:41 PM
    Having major problems with 2.3.1.  Macbooks seem to have an issue where they are connected just fine, but can't contact their gateway on occasion.  If I get 2 pings going, one to a local server, and one out to the internet, the one to the local server doesn't drop. However the one out to the internet will drop for maybe 20-30 times in a row and the magically come back. 

    These are 3 AP22s connects to a 1930 switch, with a SonicWall as a gateway. I have confirmed the SonicWall isn't blocking anything.  The same Macbooks I can connect to the ISP provided wifi with 0 issues. So it's nothing like that. 

    I don't know, these AP22s have been solid, and now they are just flaking out everywhere even though they are up to date and I've rebooted them.  I'm always at max signal on the macbooks. Real worlds transfer speeds are above 300mbps. They just suddenly can't communicate with the gateway, yet an iphone right next to it, connected to the same SSID, is sitting there running speed tests without any drops, all while the macbook can't get to the gateway.

    I've turned off Wifi6.  Not sure what else to check.


  • 2.  RE: Problem AP22 and 2.3.1

    Employee
    Posted 07-13-2021 10:24 AM
    Hi,

    can you confirm that turning off WiFi6 knob helped resolving the issue?
    Also, if you can confirm your MacOS version and Macbook type that'll be appreciated.

    Thanks,

    ------------------------------
    Jul.
    ------------------------------

    Original Message


  • 3.  RE: Problem AP22 and 2.3.1

    Posted 07-13-2021 10:36 AM
    2018 Macbook Pro.  Fully up to date Big Sur.  Turning off WiFi6 had no effect.

    I'm still testing, but so far it seems to only happen on an SSID that has both 2.4 and 5Ghz enabled. Last night I tested on an SSID with only 5Ghz, with WiFi6 ON and optimize for video streaming ON and it didn't drop at all. I'm back on a combined SSID now to verify issue returns.  If it does, next I will try 2.4Ghz only to see how that is.

    Thanks,

    Rob
    Original Message


  • 4.  RE: Problem AP22 and 2.3.1

    Posted 07-13-2021 01:10 PM
    I can now confirm that the problem exists on SSIDs with both 2.4Ghz and 5Ghz enabled. Leaving all other settings the same, simply removing 2.4Ghz from the SSID resolves the issue.  Soon I will test with 2.4Ghz only to see if that has the same issue as the combined.
    Original Message


  • 5.  RE: Problem AP22 and 2.3.1

    Posted 08-09-2021 09:35 AM
    Just adding this is still going on. I even added a 4th AP22 to rule out a hardware problem and while connected to it the problem happens. The problem doesn't happen while connected to AT&Ts provided Harris CPE. 

    It also happens on all bands, it's not isolated to 2.4Ghz. Even on a 5Ghz only SSID, with Wifi6 and media optimizations disabled, it still happens, just not as frequently.  It's obvious streaming something like HBOMax which tends not to buffer as much. You'll be watching and then it suddenly stops and says can't connect. I look at my pings and sure enough, pings to a local server are fine, but pings out the gateway drop. I can't even ping the local gateway (SonicWall) itself during this so it's not an internet problem. This persists for perhaps 15 seconds then resumes and is fine again for a couple hours usually before it happens again. idk, something is definitely wrong. It doesn't happen on wired connections. It doesn't happen connecting with WiFi to the AT&T CPE. It only happens while connected to AP22s.
    Original Message


  • 6.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 11:39 AM

    Interesting (and frustrating) problem! 

    You initially said this was only affecting MacBooks, while iPhones were okay. Does that still hold true, or have you discovered otherwise? 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 7.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:03 PM
    The problem is that it's hard to test. It can go several hours with no issue then it hits. The MacBook is used heavily for streaming video so it's obvious when it happens. None of the other devices are used in such a way where they could easily detect a drop. When the drop happens, it's usually for about 45 seconds. Enough to stop streams.  I haven't noticed it on other devices, but I can't rule it out either.
    Original Message


  • 8.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:11 PM

    Ugh. These types of issues are tough to diagnose if you can't be onsite and don't have remote monitoring. 

    We've been seeing similar issues with our provider here in Western Canada, who uses the same equipment as Comcast. It's their modem/routers (XB6&7) that are having the issues. 

    Have you double checked to be sure there's no accidental QOS / Application blocking or similar settings in the router and APs that might be choking the video feeds? Don’t forget that with an errant click you can neuter application & traffic types.

    Again - is it only Video streams that are affected? 

    Are you familiar with Wireshark? It would be interesting to fire it up when there’s an issues to see what the frames and packets are doing. 

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 9.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:22 PM
    Yes I'm familiar with WireShark, it's hard to get the timing though. I would have to sit and sit waiting.  There's no QoS or bandwidth mgmt in use. It's a synchronous 1Gbps circuit. I leave 3 pings running on the MacBook.

    1 to local gateway (sonicwall)
    1 to 8.8.8.8
    1 to a local server

    During the event. The last one, to a local server in the same subnet, that never drops. It's always fine. During the event, all pings to the local gateway, and then obviously out to the internet from there, fail. Even with QoS or BWM, it still wouldn't drop all pings for 45 seconds to the local gateway.
    Original Message


  • 10.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 04:29 PM

    All three pings are being transmitted over wifi?
    If so, I don’t think you have a wifi issue, you have a routing issue. Do you have another router you can drop in place of the sonic wall?

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 11.  RE: Problem AP22 and 2.3.1

    Posted 08-12-2021 06:39 PM
    Yes all 3 pings are being transmitted over wifi. The macbook is not wired up, it's only wifi. However, I disagree with the routing problem assessment. The SonicWall I had was a TZ400 and I had the problem. I recently replaced it with a newer TZ470 and still have the problem. There are no other routers.  All 4 APs connect to an Aruba 1930 switch which uplinks to the SonicWall. All other wired devices on the same 1930 switch have no issues, ever. It's a flat network, no subnets or VLANs. It's not the Macbook itself, as the problem never, not even once happens, while I'm connected to directly to the ISP CPEs wifi. It only happens while connected to these Aruba AP22s. It also never happened before. One of the recent updates is when it triggered.
    Original Message


  • 12.  RE: Problem AP22 and 2.3.1

    Posted 08-13-2021 02:17 AM

    You have me worried now. I'm going to pull an AP22 off the shelf and start testing. Better that we discover an issue in the lab instead of a client's in-production network. 

    If I can replicate the issue I'll report back. 

    Just thinking: The ping to your local server is over layer 2. It never drops, right? It's only the layer 3 pings & traffic that are dying. Doesn't that squarely point back to the router? WiFi only exists in layers 1&2. If the wifi was the problem, why aren’t your pings to the local server also dying? They’re literally running over the same layer 2 network.

    We know traffic is flowing most of the time - so what could be intermittently disrupting / corrupting traffic? 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 13.  RE: Problem AP22 and 2.3.1

    Posted 08-13-2021 08:00 AM
    The gateway itself is also responding at layer 2. Pings to the gateway fail. Now the ping out to 8.8.8.8 would be operating at layer 3. 

    Client IP:  192.168.1.50
    Gateway: 192.168.1.254
    Local server: 192.168.1.10

    Pings from that client, to either the gateway or local server operate at layer 2 only.  There is no routing in this scenario, it's all layer 2.  The only time layer 3 routing is involved is when I'm pinging 8.8.8.8.

    So if a ping fails to 192.168.1.254, and not to 192.168.1.10, that has nothing to do with routing, and everything to do with the APs.  That's why I disagreed with the routing issue assessment.

    I'll add that by this point, I'm using a new SonicWall and a new AP22 in the location. So neither of those are the culprit in terms of possible faulty hardware. The only other item that hasn't been replaced is the 1930 switch, but wired clients on it never have an issue, so I don't believe it is the culprit.

    What's interrupting traffic is a bug with the APs.
    Original Message


  • 14.  RE: Problem AP22 and 2.3.1

    Employee
    Posted 08-22-2021 05:51 AM
    If ping to local server 192.168.1.10, then I think it is not wifi issue. If you have same result when ping to gateway and local server, then it will be wifi issue

    ------------------------------
    DENNY .
    ------------------------------

    Original Message


  • 15.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:00 PM
    I am sorry but that's just not true. I have witnessed similar bugs on other deployments with different gear. The local gateway is the same thing as a local server. So I completely disgree with that assessment.

    I also have more notes now.

    1. The issue affects all devices. Not just Macbooks. It I have verified that Apple TVs, Macbook Pros, Macbook Airs, and a Windows Laptop all have the issue.
    2. It's not DHCP Renewal related. The issues happens even when statically assigned.
    3. I've maually channelized the network, dropping 40Mhz wide channels instead of 80, and avoiding all DFS channels. The issue still happens.

    So I'm at a loss now. This wasn't there with previous firmware.  

    Pings are consistent as well.  After running overnight:

    Pings to local server 0.5% loss
    Pings to Gateway 1% loss
    Pings to Internet 1.5% loss.

    So the loss is only 1% to the gateway but what sucks is almost all of that loss is in a chunk of 60 dropped packets in a row which breaks connections, streams, VPNs.  This happens I would say on average about once per hour to all clients connected to these 4 AP22s.  None of this is witnessed when just connecting to the ISPs Harris device.

    New firewall, new APs, none of that solved it.  All other wired machines on the same 1930 switch can ping the gateway all night long without a single dropped packet. So it isn't the wired network either. All 4 AP22s are home run to the 1930. No mesh is in use. One of these recent firmware updates triggered this.
    Original Message


  • 16.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:08 PM
    Oh and when it happens, it happens simultaneously to all clients. At least all clients on the same AP.  I need to do some more testing simultaneously from multiple APs to see if it syncs up with all APs, or if it happens individually per AP, affecting only those clients on that AP.  I know for fact that at a minimum, when it happens to 1 AP, it affects all clients at the same time, in the same way, who are associated with that AP.
    Original Message


  • 17.  RE: Problem AP22 and 2.3.1

    Posted 08-29-2021 04:27 PM
    What SonicWALL firmware are you using?

    ------------------------------
    Stephan Gladiadis
    ------------------------------

    Original Message


  • 18.  RE: Problem AP22 and 2.3.1

    Posted 08-30-2021 03:26 PM
    SonicOS 7.0.0-R906 on a TZ470.  But the problem started before I swapped out the older TZ400 that had different firmware. So I don't think it's SonicWall related at all for several reasons.

    1. The SonicWall was upgraded with a new device and new firmware, didn't fix this issue.
    2. There is never an issue on the wired network. You can ping the SonicWall at 0% loss all day from any wired endpoint including endpoints on the same 1930 switch as the AP22s.
    3. It's a flat network with a single subnet and no VLANs.
    4. There are never any issues listed in the SonicWall logs that could even remotely be related to this.
    Original Message


  • 19.  RE: Problem AP22 and 2.3.1

    Posted 08-27-2021 04:14 PM

    Thanks for the additional details. 

    We've been on vacation and unable to test this is the lab, but it's going to be one of the first priorities once we return. 


    I’m assuming no one from Aruba responds to these forums? Have you opened a ticket with support? 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 20.  RE: Problem AP22 and 2.3.1

    Employee
    Posted 09-01-2021 10:48 AM
    Hi,

    We want to assist you in getting this issue resolved. We request you create a support ticket with our Aruba Instant On support team and include setup, all client issues seen, and any additional concerns relating to the issue. This will help our team assist you in narrowing down and resolving the issue. You can reach us at: https://www.arubainstanton.com/contact-support/

    ------------------------------
    Aruba Instant On Communications
    ------------------------------

    Original Message


  • 21.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 04:45 PM
    Hi,

    I am dealing with a similar behavior here. I have 3x AP22 running, with an Aruba 1930 and a Ubiquiti EdgeRouter 4.... I am using my MacBook Air 2020 to stream using AirPlay2 over an Apple TV 4 HD. No matter what I do, it is always have dropping at an intermittent amount of time... 5 min, 1 hour.... no specific time.

    This is really frustrating. I am using this Airplay2 streaming as my 2nd monitor, and the streaming is dropping quite many times per day. I carefully followed Apple advices on how to choose right WiFi settings: channels, bandwidth, etc.

    Not much luck so far.

    I re-tested with my old all-in one wifi-router from Synology RT-2600AC alone... nothing else.... everything is stable, no issue at all. The reason I changed everything here is to let me have a better WiFi coverage.

    I am seriously considering dropping all my Aruba gears for another vendor.

    Jeff

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 22.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 05:17 PM
    That sounds identical to what I am dealing with. It wasn't always like this though, a recent firmware update caused it.  I am currently on the phone with Aruba creating a case. For me, the lengthy drops could be anywhere from 15 to 60 seconds long, and can happen anywhere from 1-4 times per hour.  I haven't been able to find a pattern to it. But I know it's not specific to the AP as I have 4 of them and they all have the issue.

    I wouldn't drop them yet. I think this is just a bug for now. I came from Ubiquiti to these and these were night and day better than Ubiquiti a first, but now the firmware updates broke them a bit.
    Original Message


  • 23.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 05:21 PM
    I'm glad we have corroboration on the issue! It sounds like the team at Aruba are listening, so let's hope they flesh out the bug and get things back on track!

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 24.  RE: Problem AP22 and 2.3.1

    Posted 09-01-2021 06:48 PM
    So far, my work around is not very gracious..... I brought back to life my old Apple AirPort Extreme wifi-router. I created a new and exclusive 5 Ghz wifi SSID... 2.4 Ghz disabled. The 5G network is running only on 1 band that I now use with my Apple TV 4 HD exclusively... for now. I also disabled the band I am using on Aruba network to avoid conflict.

    I left my MacBook Air on Aruba WiFi network to test this out. If things are not getting better, my next step would be to move the MacBook Air also on the same 5G network SSID as the Apple TV 4 HD. Problem still persist.  UPDATED

    UPDATED : Both MacBook Air and Apple TV 4 HD are now on the same 5G network using Apple AirPort Extreme router to avoid dropout.

    It was working with my old Synology too.... But I want to try also with older gears too. 

    It is indeed IMHO related to the use of Aruba AP. 

    Regards.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 25.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 10:15 AM
    Hi,

    Could someone corroborate this behavior?

    I have 1 SSID only using both 2.4 and 5 bands. The global radio settings is set to use power, for both bands, from low to high... not very high or max..... not more than HIGH period.

    From Aruba portal, I can also confirm that I did not override any radio settings for all 3 AP22.... the override checkbox for specific settings for each AP is not checked for all AP22.

    But if I take a look at each AP22 status to check which power level they are dynamically using.... I have one AP that power is set momentarily to MAXIMUM and another one to VERY HIGH.

    So my understanding is even with global radio settings threshold is set to use power level between LOW and HIGH not much, it does not seems to work at all ????

    Am I the only one?

    Regards.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 26.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 11:32 AM

    I have only experimented with a few APs' power settings, but found we had very inconsistent performance when the power was set to variable or anything less than High. Not just a small dip in performance, but a drastic reduction in throughput. 

    Have you opened a ticket w/Aruba, Jeff? 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 27.  RE: Problem AP22 and 2.3.1

    Posted 09-02-2021 12:03 PM
    Hi Ethan

    No, I did not open a case yet. I brought this issue to their attention back in July and received a few tweaks to make it work, such as using less 5G bands, etc... but that do not solved my problem. At that time, I gave up unless I found something relevant to tell them. Going forward, if they do not provide me real explanation.... I am not loosing my time.

    Like I said, I have 3 AP22.... They are all running at AVERAGE power level. But one of them is running at VERY HIGH power on 2.4 and the other one is running at MAXIMUM power on 5G band only. So far, signal/speed level are all green (good) if I refer to the dashboard. I understand that they manage their dynamic power output properly and I have so far no other issue as well.

    But my concern is "Why VERY HIGH power and MAXIMUM power are set when I asked to not go beyond HIGH only ???

    Regards,

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 28.  RE: Problem AP22 and 2.3.1

    Posted 09-11-2021 09:45 PM
    What are their dBm levels?  When I change mine the names may still say very high or max but their stated dBm value is lower.  Also there are two places to change those setting which is odd.  On the individual AP page on the bottom under radio management and then again on the inventory page under the three dot menu theirs another radio management which you would thinks changes all the APs whereas on the actual individual AP page you would think that would change just that AP but the one on the inventory page doesn't seem to change the individual pages.
    Original Message


  • 29.  RE: Problem AP22 and 2.3.1

    Posted 09-12-2021 07:50 PM
    Hi,

    I can't tell. Since my last post, I did a wifi site survey to determine a different channel for each AP on both 2.4 and 5. Then I manually setup each AP with their own channel and power level. Power level is now set MIN = MAX... same for both. So power level is always the same now. I do not let AP manage channel and power now by themselves; to many times I saw 2 AP running on the same channel.

    I still can't stream using Airplay from my MacBook Air 2020 to my Apple TV 4 HD without having disruption over time. It is frustrating..... but I cannot be sure at 100% it is related to a WiFi issue.... Airplay is always using WiFi, even if the Apple TV is wired. I will wait for a next major Apple Mac OS release soon to check if the issue still occur.

    Unless my Airplay issue, my network is very efficient and stable.

    Regard.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 30.  RE: Problem AP22 and 2.3.1

    Posted 09-16-2021 02:18 PM
    Just as an update, I opened a case a couple weeks ago. I have been testing the living hell out of this.  Aruba support felt it might be the 1930 switch causing an issue, so they moved it to local config mode, and made changes to LLDP. However, that didn't fix anything. They have reviewed everything and can't find an issue. I just got an update that they are escalating this to engineering.  

    So we aren't sure yet if it's the 1930 switch, or the APs.  All APs are connected directly to this 1930 switch. Wired clients on the 1930 switch have no issues ever and can ping the local gateway all night long and not drop a single packet. Wireless clients connected to AP22s, that are subsequently connected to the 1930 switch have this issue where they may drop as many as 80 pings in a row to the local gateway, often several times per hour, all while also still being able to ping a local server. It's really bizarre how they are denied access to only the local gateway, and subsequently the internet, but not other hosts. Again, wired clients on the 1930 switch never ever drop to the local gateway.

    Does anyone else have AP22s connected to a 1930 switch? If so, could you do some extended pings to your local firewall and see if the issue can be reproduced?  I think I might take one of my AP22s, and connect it separately to a different switch to see if it's the 1930 causing it, or the APs.
    Original Message


  • 31.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 06:03 AM
    Progress! When removing an AP22 from the 1930 switch, and connecting it instead to the upstream switch, the problem is resolved for anyone connecting to that AP. I just went 12 hours without the problem happening once whereas before, it would happen several times per hour.
    So the problem isn't isolated to the AP22s, it's somehow a combo problem when an AP22 is directly attached to the 1930 switch. Other wired clients on the 1930 switch still never have an issue. It's only when connecting via AP22, and only if that AP22 is connected to this 1930 switch. So the 1930 switch works perfectly, for anything other than an AP22. The AP22 works perfectly, as long as it isn't attached to the 1930. But put an AP22 on a 1930 and things break.
    Today I am going to try the reverse, and put a PoE unmanaged switch downstream of the 1930 switch and connect the APs to it, with the unmanaged switch uplinked to the 1930. 
    But it does appear to be some type of negotiation issue between AP22s and the 1930 switch.

    Original Message


  • 32.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 08:51 AM
    Hi,

    I am using a 8 port Aruba 1930 switch too. From the beginning, I decided to manage locally the switch instead using the cloud.... much more accessible features that are not covered when cloud managed..... Even ports problem counter is not available when using cloud management.... :-( :-(

    So far, I tried to ping my local gateway over wifi , from my laptop to another device on my network. There is a few miss (packet loss) less than 1% on a 1000 ping test. But what brought to my attention is the latency for the ICMP packet reported between my laptop for both gateway and other devices running on my network. On a test with 1000 packets sent... an average latency from 10 to 20 ms..... seems high on a « not busy at all almost sleeping network ». :-)

    ** At the time I did my test, I was using less than 500 kb bandwidth overall on my network. I have only 1 laptop running, else are IoT or tablets and phone. Network almost idle *** 

    If I do the same test from every AP to the 1930 using wired network, or from the gateway to the 1930 (wired network too).... normal report... less than 5 ms... most of the time near 1-3 ms every time. Latency seems well on target for such devices.

    So ....

    from my laptop (over wifi) and the gateway OR from my laptop (over wifi) a MacBook Air 2020 to any network device = average reported time 10-20ms with 1000 packets sent
    from laptop (over wifi) to 1930 switch = same result as above

    But ...

    from the wired AP to Gateway or from wired AP to 1930 switch = average reported between 1-3ms,

    So the WiFi is adding an additional 10-15 ms latency between the laptop and everything pinged on my network. It should be less than 5ms. Wow!!! And like I said, when pinging over wifi.... I also have an average of  0.3%  - 0.5% packet loss. NO packet loss when ping from wired devices. 

    Why I am having much latency added over wifi ??? Don't know yet.

    Regards,

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 33.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 06:17 AM
    Jeff see my post above. Can you try moving an AP22 off of the 1930 switch in some manner and see if the problem is resolved for guests on that AP? Just hook it up to some other PoE switch, or any switch and use a PoE adapter. My problems disappeared when I moved the AP22 to a different switch. Would like to see how this affects yours as well since I am sure you have the same issue, as you also have a 1930 switch.
    Original Message


  • 34.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 10:18 AM
    I am on it! Let me check....

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 35.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 10:39 AM
    Ok, this is the test I did.

    I connected 1 AP22 (powered over Power supply) on a small unmanaged switch and did again some ping tests.

    I turned off my 2 other AP22.... only 1 AP22 was turned on for my test making sure that I am using the right one connected to my unmanaged switch (not 1930).

    Results

    Over wifi

    Ping 100 packets from laptop over wifi (using AP not connected to 1930.... but using an unmanaged switch)  to gateway (wired) = average latency 11ms with less 1% packet loss
    Ping 100 packets from laptop over wifi to AP22 IP address (wired) = average latency 10ms with less 1% packet loss

    Wired only

    Ping 100 packets from AP22 wired (using web interface tools) to gateway wired = average latency less than 2ms, no packet loss
    Ping 100 packets from AP22 wired (using web interface tools) to 1930 (wired) assigned IP address = average latency less than 3ms, no packet loss
    Ping 100 packets from Gateway wired (using web interface tools) to AP22 (wired) assigned IP address = average latency less than 2ms, no packet loss
    Ping 100 packets from Gateway wired (using web interface tools) to 1930 (wired) assigned IP address = average latency less than 2ms, no packet loss


    So far, doing a ping test over wifi using an AP22 adds substantial latency... as it should be less than 5ms... not 10ms or more. Using previous results, doing ping test thru a 1930 switch seems to add more latency than using another brand name switch when tests are done over wifi.



    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 36.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 11:43 AM
    Jeff,

    The latency jumping to 10-15ms isn't going to kill your Airplay sessions. But the lengthy drops that I am describing might. If you have your AP22 connected to the 1930, then let a ping run for an hour to the local gateway, then go back scrolling through it, you may see sections of anywhere from 30 to 80 dropped packets in a row. This may result in over packet loss for the hour around 1.5%.

    Then, with the AP22 connected to a different switch instead of the 1930, do the same test. If I am correct, when scroll back through the results of the ping test, you will not see sections of 30 to 80 pings dropped in a row, and will see overall packet loss for the hour to to the local gateway at around 0.2%.

    You really need to let it run for at least an hour to be sure of a proper test.

    Rob
    Original Message


  • 37.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 01:38 PM
    Hi Rob,

    I know the 10-20ms will not kill my stream with Airplay.... but I still found such latency is unusual for such device when having a network traffic almost idle. We should see results closer to 1-3ms than 10-20ms for 1 hop.

    I started a ping test and I will let it run for a couple of hours. Packet loss might be the culprit for sure. Inside Macbook log, I recently followed the airplay message log until a disconnection..... when the disconnection occurred, error in Macbook OS log, a message was related to a connection loss between the 2 devices.

    Thanks for posting your results :-) 

    Best regards.

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 38.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 02:19 PM

    I ran a few ping tests just out of curiosity. 


    UniFi Lab: pings to gateway= 7-8ms
    AIO Lab: (AP12 + 24 Port PoE1930)= 12-18ms

    Both networks have no traffic transiting them. 

    I find it a bit odd that ping times are 50% higher with the same test device using the same basic parameters. 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 39.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 12:49 PM

    I have another update. I have confirmed that the problem is indeed a compatibility issue between AP22 and 1930 switch. I have taken an unmanaged PoE switch and tried it both upstream and downstream of the 1930 switch. When AP22s are attached to the unmanaged PoE switch, either upstream or downstream of the 1930, they work perfectly. I even used the same port on the 1930 that was previously going to an AP22, to rule out any port issues.

     

    The AP22s actually work just fine, as long as you do not attach them directly to a 1930 switch, but they can pass through it, if first connected to another switch.  I would imagine Aruba should be able to lab this up now that they have very specific on how to reproduce it. I let them know in my ticket.

     

    For now, I have to leave this unmanaged PoE switch hanging off this Aruba 1930 to power the AP22s. As longs as they are not directly connected to the 1930, they work fine. Hopefully they can figure something out soon, but at least I have good wireless again, I just can't use their switch for wifi.
    Original Message


  • 40.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 01:58 PM

    Sorry, I spoke too soon. The problem still happens if the AP22s pass through the 1930 switch at all. So attached to a downstream switch, the problem eventually happens. I timed it for an hour and it didn't happen, but right after that it did.

     

    So as long as the AP22 never passes through the 1930 switch, they will work fine. Anything else connected to the 1930 switch directly, that is not an AP22, also works fine.


    Original Message


  • 41.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 02:05 PM
    Rob, I guess you manage your 1930 locally..... I am wondering if your errors counters for each port are like mine..... unless the total amount of packets..... always no errors nor collisions..... seems strange to having all at 0..... it is normal to have a few hits and missed from time to time no ???

    Errors counters.... are they really working ?? :-)

    ------------------------------
    ====
    Jeff | VA2SS
    ------------------------------

    Original Message


  • 42.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 02:53 PM
    I have the same problem, if I move AP22 to my fortiswitch all good.
    Original Message


  • 43.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 02:19 PM
    Jeff,

    I am in local mode, they changed it from cloud to local during troubleshooting. I have errors on my port 2 of my 1930. It's the same port I use for uplink.  I have no collisions or errors on any other port, they are all 0, but I do have 33904 "Received packets with error" on port 2, the uplink.  That might be from me unhooking it a bunch and throwing cable testers and such on the uplink. I will go ahead and clear those now to see if they come back.

    Rob
    Original Message


  • 44.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 06:31 PM
    I have confirmation that Aruba Engineering is looking into it. In the meantime, they have asked that I disable LLDP completely, on any ports in use by AP22. I went ahead and disabled LLDP and LLDP-MED on every port. Will let this run for a bit to see if it fails.

    Ebakke84 yes that sounds like the issue. My AP22s work great as long as they never pass through a 1930. I can tell you though that this wasn't always the case. I had them working great for a while. I believe it was the latest 1930 switch firmware that introduced the issue, Aruba team seems to be honing in on the switch as the culprit as well, but we are still working through it.
    Original Message


  • 45.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 08:14 PM
    Well that didn't work. Disabling LLDP and LLDP-MED had no effect. Problem was still there. The only thing that works is not having a 1930 anywhere in the path of the AP22s. And still, like always, I'm having tons of pings going from wired clients on the 1930 and they never experience any issue whatsoever.
    Original Message


  • 46.  RE: Problem AP22 and 2.3.1

    Posted 09-20-2021 03:54 PM
    I went the whole weekend stressing the wireless. It didn't fail once when not using a 1930 switch. I'm thinking of scrapping this 1930, not a fan anymore.

    - It's so friggin loud
    - It no longer works with even their own APs
    - You have to switch it to local mode to get to almost anything

    idk, I'm hating on this 1930.
    Original Message


  • 47.  RE: Problem AP22 and 2.3.1

    Posted 09-20-2021 04:04 PM

    Thank you for doing all this TS'ing and sharing it with us! I haven't had Time to stress test my set up, but haven't had any reports of this issue from the field yet. That said, we've not deployed AP22s and 1930 switch combinations yet. 

    Hoping Aruba can update the FW and get things back to good. 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 48.  RE: Problem AP22 and 2.3.1

    Posted 09-24-2021 03:59 PM
    Hi,

    I know someone from Aruba is trying to get in touch with me and sent me a private message. I tried twice to answer the private message but never received an answer..... Even my sent folder still empty.... Do they have issue with their community portal too ??? :-)

    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 49.  RE: Problem AP22 and 2.3.1

    Posted 09-20-2021 04:09 PM
    Hi,

    I do agree that it is pretty annoying. So far, I do appreciate my 1930, the 8 port version, it is fan less. I find also very sad that you must manage it locally if you want to have access to all features.... AP should be that way too. I particularly do not like usage tracking from their AP's.... free private data for Aruba. :-(

    Even without the use of the 1930, I still have an issue with AirPlay and random disconnection. Loss packets, even if it is 1 out of 200 from time to time,  are present with or without the use of the 1930. When wired, 100% at all time after 24 hours continuous ping.


  • 50.  RE: Problem AP22 and 2.3.1

    Posted 09-20-2021 06:52 PM
    I would appreciate if Aruba sorts this out immediately or I'm gonna have to also pull the 1930's that I presently have installed in combination with AP22s.  I only have two of these combo's right now and one I can buy time on one cuz they're not moved in but the the other is for the CEO of the NY Philharmonic and she needs a reliable wifi networks since she is working out of her house a lot because of covid.  She does a lot of zoom meeting which is exactly the kind of use where a drop is gonna really piss her off!
    Original Message


  • 51.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 10:37 AM
    Sorry no useful input, but just to say thanks for highlighting this.

    I too am looking to jump out of the Ubiquiti ship, due to continuing instability, and was going to replace all switches and APs with Instant On. I had heard that Aruba has a good reputation for stability, which is what I need above all else, but now I'm not so sure.

    Waiting very interestedly to see the outcome of this issue before making any final decision.

    ------------------------------
    Keith Richards
    ------------------------------

    Original Message


  • 52.  RE: Problem AP22 and 2.3.1

    Posted 09-17-2021 03:03 PM

    Hey KR,

    Don't discount Aruba yet - every vendor has bugs in the code, it's how they respond to discovered issues that matter most. 

    Let's not forget - we're ahead of the curve here, we're the front runners. There isn't nearly as big a pool of users to draw off as UBNT has. 


    If Aruba is responsive, communicative and fix the issues without introducing five others, we're miles ahead of the Ubiquiti situation. 😃



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 53.  RE: Problem AP22 and 2.3.1

    Posted 09-21-2021 12:15 PM
    No useful troubleshooting info but a question:

    I was planning to purchase several AP22s and 1930s to bring internet to a set of buildings. Install date is April 2022, but there are several-month delays in hardware purchases right now. With all that said, should I hold off on purchasing until this bug is ironed out, or can I assume the fix is software not hardware and pull the trigger now?

    ------------------------------
    Aron Cowen
    ------------------------------

    Original Message


  • 54.  RE: Problem AP22 and 2.3.1

    Posted 09-21-2021 12:15 PM
    I just wanted to thank everyone following up this issue. I will have to buy some equipment soon and this is the last thing holding me back from going with a full AIO setup.

    Hopefully this gets patches soon.

    ------------------------------
    Oscar Vasquez
    ------------------------------

    Original Message


  • 55.  RE: Problem AP22 and 2.3.1

    Posted 09-21-2021 12:15 PM
    No new information, just a question: I am planning to buy a set of AP22s and 1930s to outfit a set of houses. Install date of mid spring, but with shipping delays of a few months was about to pull the trigger on buying. Should I hold off or will any fix be software not hardware so might as well buy now?

    ------------------------------
    Aron Cowen
    ------------------------------

    Original Message


  • 56.  RE: Problem AP22 and 2.3.1

    Posted 09-21-2021 02:01 PM
    Aron,

    I like the AP22s. They are pretty solid, inexpensive, and cloud managed with no annual fee. I'd still pull the trigger on those if I had to do it all over again. I would not however buy another 1930. Certainly not one with a fan in it as they are like 60db. So friggin loud. Not to mention all the actual issues with the 1930s. Skip the switch, get the APs.

    Rob
    Original Message


  • 57.  RE: Problem AP22 and 2.3.1

    Posted 09-28-2021 05:18 PM

    Has Aruba actually acknowledge this issue yet?  Usually a company with let the community know they are aware of the issue and if they were able to replicate on their test bench.  At least then we would know there are engineers actually working on this issue.  Their silence isn't very assuring that a fix might be in the works and its not very assuring that this product line is getting the attention it deserves or we deserve.  Reminds me a lot like Cisco and their SMB line and now long it took to get critical updates for basic functions to work properly.  

    If I have to pull these switches in order for the AP22 wifi networks to be reliable I will but I won't be using these switches in the future if Aruba can't even keep us informed or even acknowledge the issue and that's its been replicated.  


    Original Message


  • 58.  RE: Problem AP22 and 2.3.1

    Posted 09-30-2021 02:01 PM
    Not really. They said they were going to lab it up with Engineering but I just got an email today asking me to do even more tests. I don't even have the APs on the switch anymore. At first it sounded promising but I can't get clear information from them. Sounds like they just do enough to kick the can down the road.
    Original Message


  • 59.  RE: Problem AP22 and 2.3.1

    Posted 09-30-2021 03:04 PM

    Sounds like the the D team engineers get this line like with Cisco and the A,B,C engineers deal with the enterprise gear which I get but….  I do have some AP22's on the shelf but not sure about a 1930 switch.  I do have a couple operational combo's in the field that I could play with remotely but unfortunately I didn't use a VPN capable router cuz I consider these budget jobs so I just use the ISP's.  Now if we had more capability with the web portal, maybe even if the portal allowed a telnet connection we could do things with command line prompts.  

    I guess I should see what 8 port POE switches I have in stock and plan on a road trip.  Fortunately while the two jobs are over an hour away they're within minutes of each other.


    Original Message


  • 60.  RE: Problem AP22 and 2.3.1

    Posted 10-05-2021 11:14 AM
    Same problem here. It would be nice if someone from the Aruba team would respond to this thread.​

    ------------------------------
    Oliver Meyer
    ------------------------------

    Original Message


  • 61.  RE: Problem AP22 and 2.3.1

    Employee
    Posted 10-06-2021 05:14 AM
    The Aruba Instant On team has been working closely with the user who has raised the issue and is continuing to work with them. 
    Despite trying to reproduce the issue, so far we have not seen this in-house. 
    We want to assure customers and partners that Aruba will continue looking at this to validate this issue.


    ------------------------------
    Aruba Instant On Communications
    ------------------------------

    Original Message


  • 62.  RE: Problem AP22 and 2.3.1

    Posted 10-21-2021 09:29 AM
    I would disagree that Aruba has been working closely with me. Replies are few and far between. Almost all of the testing has been put on me, and I don't have the time to continually test, breaking my wireless in the process. No one even replied to me the results of internal testing or asked me for any input at all in the methodology. In fact, this is the first I've even heard that internal testing even began. It's pretty clear there is an issue with the hardware/firmware on the switch. I can run anything through it (that I have tested) with zero issues, except for Aruba AP22s. I'm losing faith in this product. I have requested an RMA at this point to see what that does as I have nowhere else to go except to sell this switch on ebay, hope it works for the next guy, and move on. To anyone else, stay far away from Aruba switches if you plan on using them with Aruba APs.

    Rob
    Original Message


  • 63.  RE: Problem AP22 and 2.3.1

    Posted 10-21-2021 10:18 AM

    Well, let's take a moment to acknowledge that you were 100% certain that it was the AP22s, and you dragged them through the mud, so to speak. Now it's the switch. 

    It's understandable that you'd be frustrated, but your constant negative attitude and words are off putting. 

    Of course you have to do more testing - if they can't replicate the issue in the lab they have to go to the source of the problem. Isn't that pretty standard protocol for Cisco and Ubiquiti too? 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 64.  RE: Problem AP22 and 2.3.1

    Posted 10-21-2021 12:39 PM
    Let's take a moment to acknowledge that this is 100% an Aruba issue, that exists only between Aruba APs and Aruba switches.
    Let's take a moment to acknowledge that Aruba couldn't make any config changes to their equipment to resolve it.
    Let's take a moment to acknowledge that I have already spent countless unpaid hours on this.
    Let's take a moment to acknowledge that sure, I first thought it was the APs, but it was still an Aruba issue, as it ended up being the Aruba switch.
    Let's take a moment to acknowledge that I tried to avoid issues like this by staying with the same vendor for APs and switches.
    Let's take a moment to acknowledge that they haven't even asked me for guidance on internal testing, or even shared any results with me, or even let me know how/when/what they were testing.
    Let's take a moment to acknowledge that my time is valuable, and I shouldn't have to waste so much of it on a single switch and 4 APs from the same vendor.
    Let's take a moment to acknowledge that others are having the same issue.  

    Let's take a moment to acknowledge that you putting this back on me and saying I have a bad attitude is off putting.  Peace.
    Original Message


  • 65.  RE: Problem AP22 and 2.3.1

    Posted 10-21-2021 12:44 PM
    Oh a couple more:

    Let's take a moment to acknowledge that the Aruba switch works perfectly fine without the Aruba APs. 
    Let's take a moment to acknowledge that the Aruba APs work perfectly fine without the Aruba switch.
    Let's take a moment to acknowledge that the problem only happens, when you combine the Aruba APs with the Aruba switch.

    That should cover it.
    Original Message


  • 66.  RE: Problem AP22 and 2.3.1

    Posted 10-22-2021 07:29 AM

    I still can't explain WHY when pinging over wifi when using an AP22 and a 1930 results with packets loss. Pinging over wired connection, no issue.

    Aruba, you should begin in your lab by testing this one first.


    And for the sake of this group….. then give to us your results HERE. 

    Regards. 




    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 67.  RE: Problem AP22 and 2.3.1

    Posted 10-22-2021 03:38 PM
    Hi,

    To Aruba guys... one more thing...

    When trying to keep alive my local browser sessions with my 1930.... default value is 10 minutes before disconnection..... even if I set it at 60 minutes... it will disconnect at 10 minutes....


  • 68.  RE: Problem AP22 and 2.3.1

    Posted 10-22-2021 07:02 PM
    That's some solid troubleshooting Jeff! I wonder if the Unicast Control rate is set too aggressively by default in this firmware update? Strange that it's only AP22s that trigger the issue. Hmm.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 69.  RE: Problem AP22 and 2.3.1

    Posted 10-23-2021 08:32 AM
    Still testing…


  • 70.  RE: Problem AP22 and 2.3.1

    Posted 10-24-2021 09:51 AM

    I unboxed a 1930-24 today just to look at the local interface, never have and on my box global storm control was off and on individual ports unicast storm control was off while broadcast and multicast were on.  Was this the case in your box or did Aruba change the default config overnight so when my box came online and updated unicast storm con is now off on all ports.  


    There really needs to be the ability to have at least the switches both cloud and locally managed.  It would be nice if we could also have full access through the cloud for those times when we don't also install a vpn router.  


    Now if this was a recent change did they also change already deployed switches?  Turn off unicast until they sort out the issue?  Would be nice if I could remotely look and see and change if I need.


    Original Message


  • 71.  RE: Problem AP22 and 2.3.1

    Posted 10-24-2021 12:43 PM

    Bad news! The disconnection issue still occurs randomly, even if last Friday, I had no disconnection for all working day.



  • 72.  RE: Problem AP22 and 2.3.1

    Posted 10-24-2021 01:35 PM
    I have replaced 1930 with fortiswitch, and also tested with cisco CBS250, all good and no more packet loss. (macbook air m1 and apple tv as clients)

    aruba, please fix
    Original Message


  • 73.  RE: Problem AP22 and 2.3.1

    Posted 10-24-2021 04:29 PM

    Thanks for your input! II Looking forward to receive my new switch and test everything again. 


    Regards. 



    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 74.  RE: Problem AP22 and 2.3.1

    Posted 10-25-2021 10:20 PM
    Yes I am frustrated. I am frustrated that so many manufacturers put out equipment, that has obvious issues. They skimp on solid testing, and put out issues, then expect us to test them for them. This is an attitude that has become so prevalent anymore. I own a 20 person MSP and I get burned out on issues like this. It's cheaper for me to throw the $300 switch away than it is to spend a day performing test after test, documenting, relaying info, etc. Sure I get it for complicated scenarios, troubleshooting SAN performance in densely populated virtual environments, whatever. But for a simple setup of a couple of APs and a switch from a single manufacturer, and you just want basic functionality to work, when you didn't buy off the shelf stuff at Wal-Mart, well it gets old. 

    Anyways, I am starting to think there are more issues with the switch, and not just with AP22s. I actually don't think there is anything wrong with the AP22s any longer, and I am 99% sure it's the switch alone. Windows PCs on the switch are perfect all the time. Wired Apple TVs (6th gen) of which I have 4 seem to also drop periodically. They drop streams for a 5-60 seconds on occasion when the internet is rock solid. I can't ping out from them so they are harder to troubleshoot. Just like Jeff and Ebakke84, I am about to pull this switch completely from the network to see if it stops that as well. I already resolved all of my wireless issues by not passing the APs through it, so now it's time to move all traffic off of it I guess.

    I'll sit on the switch for a bit but if this isn't resolved soon it will have to go.
    Original Message


  • 75.  RE: Problem AP22 and 2.3.1

    Posted 10-27-2021 01:20 PM
    I can confirm that removing the 1930 switch entirely from my network has resolved every single network issue I have seen, including Apple TV streaming. Not one hiccup since. I removed the switch yesterday morning, and everything has been perfect since. Aruba is sending me a replacement 1930 switch to try, but I really think it's firmware and not the switch, since it used to work fine and problems started after firmware updates. Not to mention others having the same issue which points to faulty firmware, again since it worked in the past. I can't say for certain the cause on the switch, I can only say that the 1930 switch is 100% the culprit and removing it fixes it all.
    Original Message


  • 76.  RE: Problem AP22 and 2.3.1

    Posted 10-27-2021 08:58 PM
    PROBLEM SOLVED..... AT LAST !!!!!!!!!

    I finally found the issue we have..... holly smoke.... I have been working on it since last June! Let me explain....

    SETUP 1 for test

    Device 1 = MacBook Air 2020 over WIFI
    Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
    Device 3 = Apple TV 4 as my second display using AirPlay
    Ubiquiti EdgeRouter 4
    Aruba AP22 (1 have 3 of these)
    Aruba 1930 8 port

    Problem with SETUP 1

    • Ping 5000 packets from device 1 to device 3 or device 3 to device 1 ====== packet lost +/- 1%
    • Loosing AirPlay connectivity on a random basis between when streaming from device 1 to device 3
    • But..... Ping 5000 packets from device 2 to device 1 or device 2 to device 3 ======= NO packet lost..... Huuuummmm why!?!?!?!
    • Then I tried with a different setup.

    SETUP 2 for test

    Device 1 = MacBook Air 2020 over WIFI
    Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
    Device 3 = Apple TV 4 as my second display using AirPlay
    Ubiquiti EdgeRouter 4
    Aruba AP22 (1 have 3 of these)
    Aruba 1930 8 port replaced by a TP-LINK 8 port POE switch

    Problem with SETUP 2

    SAME results as SETUP 1.... packet lost with device 1... loosing connectivity streaming using AirPlay from device 1 to device 3 !!! 
    • But..... Ping 5000 packets from device 2 to device 1 or device 2 to device 3 ======= NO packet lost..... why why why!?!?!?!

    • Then I tried with a different setup.

    SETUP 3 for test

    Device 1 = MacBook Air 2020 over WIFI
    Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
    Device 3 = Apple TV 4 as my second display using AirPlay
    Ubiquiti EdgeRouter 4 replaced with an old Apple AirPort Extreme router
    Aruba AP22 (1 have 3 of these)
    Aruba 1930 8 port replaced by a TP-LINK POE switch

    Problem with SETUP 3

    NO PROBLEM AT ALL. No packet loss with device 1 and no AirPlay disconnection between device 1 and 3..... WHY ?!?!?!?!

    • Then I tried with a different setup.

    SETUP 4 for test

    Device 1 = MacBook Air 2020 over WIFI
    Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
    Device 3 = Apple TV 4 as my second display using AirPlay
    Ubiquiti EdgeRouter 4 replaced with an old Apple AirPort Extreme router
    Aruba AP22 (1 have 3 of these)
    Put back my Aruba 1930 8 port

    Problem with SETUP 3

    NO PROBLEM AT ALL. No packet loss with device 1 and no AirPlay disconnection between device 1 and 3..... WHY ?!?!?!?!

    My EdgeRouter has IPV6 disabled .... never learn IPV6... a guy from old school.... I setup my router with command line... I said I am an old school guy. :-)
    My old Apple AirPort Extreme has IPV6 enabled ..... or a clue here!
    My MacBook has IPV6 enabled
    BUT the Raspberry Pi always worked... never had ping issue or packet loss at any time..... no IPV6 at all as well was set on this device.

    Remember, all people were complaining when using a MacBook Air or other Apple computer.

    So I came up again with the SETUP 1....... the original setup I REALLY WISH TO USE !!!! 

    What did I do that solve my problem? On the MacBook Air, I turn off COMPLETELY protocol IPV6 by using this command : networksetup -setv6off Wi-Fi

    See here for more info : https://pall.as/ipv6/ipv6-on-mac/

    I suspect here IPV6 from the MacBook was bringing network confusion ... using IPV6 momentary instead of IPV4.... when broadcasting over the network or dealing with the Apple TV AirPlay steaming.... for an unknown reason.... I can't explain.... like I said, I never learned or used IPV6 at all..... an old school guy back in the 90'. :-) And because my EdgeRouter does not have IPV6 setup.... no answer from the gateway on IPv6.

    I also disabled IPv6 support on the 1930.

    So the end of my story..... if you have devices (especially Apple computer) with IPV6 enabled and not have a complete IPV6 network properly setup.... especially setup and full handled with your router..... Just disable it entirely.

    So now with SETUP 1.... I have no more, I said NO MORE, issue since IPV6 has been disabled on my MacBook Air 2020 using the command line.

    End of my story and case closed.... as I am concerned.

    Jeff

    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 77.  RE: Problem AP22 and 2.3.1

    Posted 10-29-2021 10:26 AM
    I always disable IPV6 on all my routers, I don't understand it either.  Since disabling it on hosts isn't practical on systems we install that means we're gonna have to learn it and properly config our network switching and routing devices or this problem will persist.
    Original Message


  • 78.  RE: Problem AP22 and 2.3.1

    Posted 10-29-2021 10:52 AM
    Hi VAV, usually, you don't have to deal with..... Without any issue or weird behavior when using Windows or Linux..... it seems a specific issue with Mac OS... even latest version...... weird issue.... but at least everything now works. Same test with iOS or iPad is also good to go.... only Mac OS as of now.

    It was NOT an Aruba issue at all at the end.

    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 79.  RE: Problem AP22 and 2.3.1

    Posted 10-29-2021 11:48 AM
    I started intentionally turning it off on my cisco routers just cuz there was supposedly performance hits or maybe it was log errors that would pop up as the ISP played with IPV6 on their end.  Supposedly in my neck of the wood they haven't implement it yet either so why enable something that's not needed.  It's not need at all on a LAN AFAIK so why apple would do something screwy is beyond me but they have a history of causing network problems as they think they're "pushing the boundary" of technology.  

    I also had a setup and I didn't see any issue using a 1930-24G with AP22's but all I had laying around were older APTVs boxes, 1080p models with a Firewalla Gold that did have IPV6 disabled by default.  FYI, Firewalla is an interesting router but again app based cloud logins so I'm trying to like it.  Giving it a shot anyway along with Mikro-Tik and Zyxell since I'm tired of cisco SMB, hence Aruba APs and switches.
    Original Message


  • 80.  RE: Problem AP22 and 2.3.1

    Posted 10-29-2021 12:19 PM
    Perhaps your issue was different Jeff. I have IPv6 enabled all over, including my SonicWall TZ470 firewall. You really shouldn't disable IPv6 on windows machines, especially servers so I never disable it. Anyways, I waited a couple more days to respond to be 100% sure what I am saying holds. It does. Since pulling the 1930 switch and replacing it with dumb trendnet switches, all of my problems have disappeared. Not one hiccup on wireless or Apple TVs. I didn't have any fancy config on the 1930 switch either, not even any VLANs. They did take it out of cloud mode during troubleshooting to disable LLDP and such but none of that had any effect.

    I have a new 1930 coming, so I will hook it up leaving it as close to default as possible just to see. I'll report back after letting that run for a few days once I get it. They are still trying to source one though so it sounds like it might be a bit.
    Original Message


  • 81.  RE: Problem AP22 and 2.3.1

    Posted 10-29-2021 01:38 PM
    I'm not saying to systematically disable all over the place IPv6..... I discovered after all  tests my Raspberry has still IPv6 enable and never had any ping loss.... even with the 1930 into the equation..... but this is not the case with MacOS.

    Disabling IPv6 on Mac OS only is a work around.... only a work around.... that solved 100% my problem..... even without the 1930, I was having issue. I you take a look a all different setup I used in my tests, you should see that the 1930 is not where we should look at..... anyway.... you may have something different at your location that it is different from here.... If I wish to keep IPv6 on my MacBook enable, I have just to turn on IPv6 on my Ubiquiti router..... which I do not have at the moment..... When I was using my old Apple router with IPv6 enabled.... everything was just fine.... even with 1930 and AP22.....

    Anyway...

    Good luck. :-)

    ------------------------------
    ====
    Jeff
    ------------------------------

    Original Message


  • 82.  RE: Problem AP22 and 2.3.1

    Posted 10-22-2021 07:44 PM

    Sorry if I came across as rude, but you make such definitive statements when time and methodical troubleshooting is required. You also seem so incensed that there's an issue with a single AP/switch combination. 

    I'm sure you've given much more grace to Ubiquiti in the past, and this is a new product line (albeit based on established lines and from a solid manufacturer)…. there's going to be the odd issue. 

    I am sorry you've had to go through all that and I do sincerely thank you for bringing this to everyone's attention. 

    I just find that one should be able to navigate these issues without being derogatory and negative. Maybe you just need to come here and vent,  it not sure it's warranted. 

    I often wonder if issues don't get resolved as quickly for some due to their negative attitude. You can catch more flys with honey than vinegar, ya know? 

    At any rate, thank you. 



    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 83.  RE: Problem AP22 and 2.3.1

    Posted 10-22-2021 04:54 PM
    Good luck to those that have already bought into this option. But it really doesn't look like it's the reliable alternative I'd been led to believe it might be. Thanks at least to Aruba for providing this forum to allow the issues with their products to be publicised, and to warn prospective new purchasers away.

    ------------------------------
    Keith Richards
    ------------------------------

    Original Message


  • 84.  RE: Problem AP22 and 2.3.1

    Posted 11-10-2021 11:49 AM
    As an update, they finally found a new switch for me and it shipped out. I don't have it yet. However, in the meantime, the network has been 100% rock solid for me without that switch. It's been nice not having wired Apple TVs freeze/buffer, or drops on the wireless ipads and macbooks. So nothing has changed with my ID of the problem being the switch. I don't disable IPv6 anywhere, and am still using a SonicWall TZ470 for a firewall. I'm just using some dumb trendnet switches at the moment without a single problem. Still can only point at the switch and say that somehow, some way, it's the culprit.
    Original Message


  • 85.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 01:41 PM
    New switch installed. Completely bone stock. Not 1 configuration change. One thing I noticed was that my 1930 was a hardware version 1.00 and this one is a 8.00. Maybe that will help and it was a hardware bug? Not sure yet. Right now, things are good, but it's only been 30 minutes. Will report back if anything changes.
    Original Message


  • 86.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 06:00 PM
    Fingers crossed for you! Let's hope it was a Hardware / Firmware issue that's gone and it doesn't come back to visit.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 87.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 07:26 PM
    Still broken. Exact same as before. I have even simplified it. 4 APs, going to a single 1930, which is completely default and uplinked via port 1 directly to a SonicWall TZ470. I still get drops to the gateway. During this time, link between the SonicWall and the 1930 is fine. I will get a 60 second long drop, where the wireless client cannot ping the gateway. However, it can ping anything else, including the AP and the switch. And during this failure, the SonicWall can ping the switch, and the AP, perfectly fine. Link is maintained between the 1930 and the SonicWall. SonicWall can always ping the switch the and the AP. It's just that the wireless clients can't get to the gateway via an AP.

    And again, taking this completely bone stock 1930 out of the equation and replacing it with an unmanaged switch, and boom. Perfect wireless. I guess I will report back to support.
    Original Message


  • 88.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 07:28 PM
      |   view attached
    This is what is looks like from the MacBook that is associated with the AP at 10.100.1.41.
    Original Message


  • 89.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 08:24 PM
      |   view attached
    Here's a quick and dirty diagram. Note how to even keep pinging the file server above at 10.100.1.2 from the Macbook, you must traverse the AP22, then the 1930, pass through the built-in switch on the SonicWall, and down to the file server. That entire path never drops. But pings directly to the SonicWall drop. Pings out of the SonicWall to anywhere other than the client, during this event, including the 1930 switch, the AP22, are perfectly fine and link is never dropped. SonicWall logs are clean, it's not blocking anything.

    So the only thing I can say is that the during this event, the SonicWall can't ping the client. The client can't ping the SonicWall. Both the client and the SonicWall can ping anything else (except of course the client can't ping out past the SonicWall to the internet as well).

    I can't explain it, but it's like the switch, or the AP, or the combo of them, is somehow blocking access to the gateway during this event. Even though traffic can pass right through the builtin-switch to get to a file server.  Keep everything the same, and change out the bone stock 1930 with a dumb trendnet switch and the problem disappears.

    It doesn't even happen to all clients at the same time. I have just witnessed the ipad drop, where the macbook  did not, and they were both associated with the same AP.
    Original Message


  • 90.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 08:38 PM
    It's absolutely nuts that I can sit here with a Macbook and a brand new M1 iPad Pro 13", both associated with the same AP, and one will drop, and the other will not. They don't even drop at the same time, but at random different times, even though they are both on the same AP. They both will drop, but not at the same time.

    I don't know how else to prove that is something logical. You could make a case saying it's the SonicWall, but why doesn't the SonicWall have any issues then when I change out the Aruba switch with a Trendnet?
    Original Message


  • 91.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 09:57 PM
      |   view attached
    I guess tomorrow I am going to change it to look like this. This way only 1 AP is going through it so I can leave it up for testing and log gathering. What's even more annoying, is that you can't limit SSIDs to certain APs. So I have to just put my wireless testing gear next to it and hope they don't associate with any other APs. It would be nice if we could scope down the network, to choose which APs each network is broadcast on.
    Original Message


  • 92.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 10:12 PM
    Yeah, aruba really should make that feature available.
    This is puzzling to me. By adding in a managed switch, you're adding Layer 2 features, with the possibility of some additional Layer 3 restrictions. Hmm...

    - In your controller, do you have ~any~ VLANs set up?
    - Does the Sonicwall broadcast any additional Ip ranges/subnets? Because this feels to me like an IP / subnet conflict somehow.
    - What about the "Network Access" tab under your wireless networks? Is it Unrestricted? 
    - Have you tried creating a new Site in the controller? Starting fresh may help weed out any issue that may be hiding in the code somewhere. That would follow you no matter what device you have.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 93.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 10:19 PM
    No VLANs at all. Not on the SonicWall or the 1930. The 1930 is bone stock. I unboxed it, added it to my account, racked it up and plugged in the APs. I didn't even set a static on it, just did that via DHCP reservation.

    No other IP ranges in use anywhere, SonicWall or otherwise. 10.100.1.0/24 is the only range in use.

    Network access is set to Unrestricted. Wi-Fi6 is off. Streaming optimizations are on. I haven't seen any of these settings here make a difference though. I have tried every combo.

    I haven't tried a new site yet. I guess I could do that. Seems like a big pain in the ass. Also, the switch still did this even when managed locally, removing the cloud site from the equation.

    Thanks for the suggestions.
    Original Message


  • 94.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 10:21 PM
    Oh and to be crystal clear, I didn't "replace" the switch in the portal with the old one. The other one was already gone since we changed it to local trying to fix it. This new one was set up as a new switch. So it's completely default as it would come out of the box, outside of being attached to an account.
    Original Message


  • 95.  RE: Problem AP22 and 2.3.1

    Posted 11-15-2021 10:27 PM
    After 15K pings.  Pings to the fileserver at 10.100.1.2 have a 0.1% loss. Normal and expected.
    Pings to the gateway at 10.100.1.254.  2.4% loss. Totally not normal or expected and almost all of the drops are in those contiguous blocks that last up to around a minute, sometimes a little longer, sometimes a little less.
    Original Message


  • 96.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 04:42 PM
    Well this sucks. So I pulled the switch out of it's normal rack location. I set it up right next to the SonicWall in my office, patched directly to it, with nothing else on the switch but a single AP22. This is right next to me, and I've been on it non-stop from my Macbook and ipad.

    At first I was dumbfounded. It wouldn't fail. It worked properly all night last night. I then again tested the Cat6a drop that separates them normally (which is the only difference in this config), and it again passed. I scratched my head and went to sleep. Started using it again this AM. All day today it was also fine. Then, out of nowhere, and this time simultaneously, a long 60 second drop to the gateway, from both the macbook and the ipad. I immediately confirmed they were still on that same AP and they were. So this took 24 hours to show up after a fresh reboot on the switch and the AP. All patch cables are brand new Cat6a and have been replaced multiple times, and verified, just to be sure. I don't know how else I could simplify this.

    So this shows that a full reboot of the switch and AP can let things work for a while. This time it took 24 hours after a fresh reboot before it started showing up again. I'll bet it will now continue into the night. Perhaps a memory leak or something in the firmware?
    Original Message


  • 97.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 05:03 PM
    Oh, that does suck - I'm sorry. Was hoping it was going to be good for you. 
    Question; is your SSID combined 2.4 & 5GHz or are they separated?
    Also, why don't you turn "Optimize for video streaming" Off? (Have you troubleshot with it off?) Those are exactly the type of "optimizations" that can create issues.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 98.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 05:52 PM
    The main SSID is combined. But I have also tested with only 2.4 or only 5 and they still had the same issue. Also, I have tested all options as I have stated above. Optimizations on/off, 2.4 only, 5 only, combined, WiFi6 on/off.  I have spent so many months on this I think I am about to give up and just sell it lol. 



    Original Message


  • 99.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 06:03 PM
    I wonder what the trigger / issue is? Since I tested and posted, I've decommissioned our UniFi network and have been using the AIO AP22 + AP15 exclusively without issue. What's different with my set up compared to yours and others? This is what has me scratching my head. 

    I keep running ping tests for hours on end to see if I can catch this elusive issue, but I just haven't seen it. So weird. Can't blame you for throwing in the towel.

    Ping tests to local and external


    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 100.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 06:23 PM
    The only difference is that I am running a TZ470 and you are running a Ubiquiti. Maybe there is just some sort of weird incompatibility between SonicWall and Aruba. I only have 2 patch cables here. AP to switch, switch to SonicWall. 1 cable is 7', 1 is 3' and both are new and have passed verification tests. I'm sitting right next to it. That leaves nothing other than the firewall as a difference if you are also using a 1930 switch.

    Also, what app is that you are using for charting icmp? It looks nice.
    Original Message


  • 101.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 07:23 PM
    I'm using a Netgate SG-1100 (pfSense) but same thing... it's a different gateway. I wonder if it's somehow tied into having multiple AP22s? I'm trying to recall if Jeff from the other thread had multiple AP22s. I know he said disabling IPv6 seemed to do the trick, but I've left everything default, and haven't had the issue.
    Maybe I'll add another AP22 to the mix and see if that changes things. I think I'll run Wireshark to get a baseline for existing traffic first. 

    The ping program is decent, it's called ICMPUtil but it's for Mac only, made by a company called ermitacode. They make a few decent Mac networking apps.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 102.  RE: Problem AP22 and 2.3.1

    Posted 11-19-2021 07:53 PM
    Thanks I just bought it. Cool little app for $7. I'm a sucker for fancy looking dark mode graphs. I've only recently switched over to daily driving a ma so still getting used to it. Love it so far though. Now I just need my new 16" M1 Max 10/32 core, 64GB to show up. Early December supposedly. Can't wait for that bad boy.

    Yeah idk. I can't figure this out. I really have no idea why it works for you. I simply can't make it work reliably no matter how much I simplify things.
    Original Message


  • 103.  RE: Problem AP22 and 2.3.1

    Posted 11-21-2021 11:25 AM
    I now have a brand new aruba site, with only the switch and one AP22 in it. I have this switch directly uplinked to the SonicWall, in its own VLAN now, with its own SSID, and isolated from the 3 APs left in the original site. I'm going to leave it like this for a while. Unfortunately I'll be out of town tomorrow through Friday so won't be able to test during that time. 

    When I get back, I'll be hammering on just this new site. If it never fails after several days, I'm going to add 1 more AP to it. I'm just trying to get a basic configuration that works, that I can then add things to until it breaks. Since it works for Ethan with a single AP, I'm trying to replicate just that with a new site and see if that works for me also. Maybe it is related to multiple APs, or maybe my site in the cloud was just corrupted somehow. Either way, I have a new site and 1 AP only now. It's been solid for 10 hours, but it went for 24 the other day before flaking out. 

    The other half of the netowrk and the original site continues to work splendidly on the trendnet switches.
    Original Message


  • 104.  RE: Problem AP22 and 2.3.1

    Posted 11-21-2021 09:06 PM
    Well, it started up again. It's not as frequent, but even with a brand new config, and only 1 AP, I just had a long 1 minute drop to the gateway. It's almost at the 24 hour mark now. So it seems to take a while before it happens. There is no way for me to further simplify this. It's just broken.
    Original Message


  • 105.  RE: Problem AP22 and 2.3.1

    Posted 11-28-2021 12:22 PM
    I have finally achieved a baseline where it works. Take it out of cloud mode, and turn off each and every single feature on the switch. Anything that is turned on, turn off. If I do that, it works and I only have a 0.3% packet loss to the gateway over 48 hours, and never more than 2 dropped pings in a row. 

    Now I am testing local mode, but then resetting to defaults after moving to local. Hopefully I can eventually narrow down what setting is causing it.
    Original Message


  • 106.  RE: Problem AP22 and 2.3.1

    Posted 11-28-2021 01:37 PM
    Glad you've found that baseline, but sorry you're still after it!  So low and behold the issue has reared its ugly head on our network as well! 
    In our case, it's only 2.4GHz connected devices on an AP22 that are causing the issue. I have not witnessed the problem on 5GHz after a couple weeks of real time usage and occasional testing during that time.

    I'll update my testing thread and ping Jessica (JM78) at AIO to see if they want a case opened. Are you sure you've seen the issue on 5GHz as well, FL0? My hunch is this would come into play on 2.4 only & combined networks, but not on 5GHz only.

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 107.  RE: Problem AP22 and 2.3.1

    Posted 11-28-2021 02:34 PM
    I have had issues on all network types. 2.4, 5, and combined 2.4 and 5. That was one of the first things I tried way back when. Right now I am testing combined 2.4 and 5. The issue hasn't occurred on a combined SSID when the switch is in local mode with everything off. It also hasn't occurred so far with local and then defaulting the switch after changing to local to clear out any cloud settings, but it's only been a few hours like that. I plan on keeping it like this for several days, or until it fails, and if it fails, start turning things off one at a time, leaving for several days in between each change to verify it.

    That's unfortunate that you are now also seeing the issue, but it does make me feel a tad less insane. At least I have a way that works so that I don't need to sell the switch. I'll keep updating as I find things that work or don't work.
    Original Message


  • 108.  RE: Problem AP22 and 2.3.1

    Posted 11-28-2021 02:50 PM
    Yeah man, it's frustrating for sure. Based on your experience and that of Jeff's, it seems to be a software issue. I didn't have the benefit of testing our AP22s prior to you reporting the issue, so I'm blind on performance prior to 2.3.1

    Here's hoping aruba sorts it out and pushes a firmware update soon!

    ------------------------------
    Ethan
    ------------------------------

    Original Message


  • 109.  RE: Problem AP22 and 2.3.1

    Posted 11-29-2021 02:29 PM
      |   view attached
    Well it took 24 hours, but it finally just failed. I was really hoping it would work with local defaults. This is with removing it from cloud, and then setting it to defaults. Time to start shutting stuff off features one by one I guess. This will take forever having to leave it for 48 hours in between each change.

    Original Message


  • 110.  RE: Problem AP22 and 2.3.1

    Posted 12-01-2021 10:34 AM

    I just went 48 hours with no issues with the following config:

    1. Remove from cloud
    2. Reset to local defaults
    3. Set static IP
    4. Set time server
    5. Turn off "Link Flap Prevention" under Switching > Port Configuration
    6. Turn off "Auto Port Power-Down" under Switching > EEE Configuration

    Those settings seem to work. Today I have re-enabled "Link Flap Prevention" to continue testing. I think the problem might be with the "Auto Port Power-Down" setting or something related to EEE. My SonicWall is listed as having "Link Partner EEE Support" on the interface on the switch, but the APs do not show that.

    Maybe something with EEE is causing this?

     - Rob


    Original Message


  • 111.  RE: Problem AP22 and 2.3.1

    Posted 12-03-2021 03:44 PM
    It's looking like I'll never be able to use cloud mode, so I went ahead and upgraded from 1.05 to 1.07 firmware which is for local mgmt only. I have all EEE stuff disabled.

    I do have some other settings as well:

    - Static IP on it instead of DHCP with reservation
    - Disabled PoE on ports that don't need it
    - Disabled Flow Control and Link Flap Prevention Everywhere
    - Disabled Auto Port Power Down and Low Power Idle (EEE)
    - Enabled Storm Control, Loop Protection, BPDU Guard
    - Left enabled all LLDP and LLDP-MED settings, as they were by default


    So far this has gone 24 hours with no major blips. 0.2% packet loss to the gateway which is acceptable. Going to leave it like this. If it keeps working fine, I'm just going to accept this as the solution and move on with my life.
    Original Message


  • 112.  RE: Problem AP22 and 2.3.1

    Posted 12-06-2021 01:58 PM
    I have been able to achieve almost 0 packet loss. About 10 lost frames over wireless, after all night of pinging. Latency is lower as well, with avg latency of 7ms over wireless to the gateway.

    It's quite simply actually.

    1) Switch to local mgmt
    2) Upgrade to 1.07 firmware
    2) Disable quite literally every single thing on the switch. If a feature is turned on, turn it off. This includes disabling LLDP and LLDP-Med on all ports.
    3) Disable PoE on ports where you don't need it.

    Congrats, you now have something that will not drop frames, and have relatively lower latency than before.

    I will eventually play around with slowly turning one feature back on at a time, and leaving for at least a week to verify, but I'm not doing that now. I need a break from this so am signing off and will probably just close my ticket with support. If I do ever have solid findings, I will post back here, but not until I am completely done with them.

    I won't be selling any of these however. They need a lot of work before I would ever consider doing that. APs yes, switches no. That's how we are moving forward.
    Original Message