PROBLEM SOLVED..... AT LAST !!!!!!!!!I finally found the issue we have..... holly smoke.... I have been working on it since last June! Let me explain....
SETUP 1 for test
Device 1 = MacBook Air 2020 over WIFI
Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
Device 3 = Apple TV 4 as my second display using AirPlay
Ubiquiti EdgeRouter 4
Aruba AP22 (1 have 3 of these)
Aruba 1930 8 port
Problem with SETUP 1
- Ping 5000 packets from device 1 to device 3 or device 3 to device 1 ====== packet lost +/- 1%
- Loosing AirPlay connectivity on a random basis between when streaming from device 1 to device 3
- But..... Ping 5000 packets from device 2 to device 1 or device 2 to device 3 ======= NO packet lost..... Huuuummmm why!?!?!?!
- Then I tried with a different setup.
SETUP 2 for testDevice 1 = MacBook Air 2020 over WIFI
Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
Device 3 = Apple TV 4 as my second display using AirPlay
Ubiquiti EdgeRouter 4
Aruba AP22 (1 have 3 of these)
Aruba 1930 8 port replaced by a TP-LINK 8 port POE switch
Problem with SETUP 2
SAME results as SETUP 1.... packet lost with device 1... loosing connectivity streaming using AirPlay from device 1 to device 3 !!!
- But..... Ping 5000 packets from device 2 to device 1 or device 2 to device 3 ======= NO packet lost..... why why why!?!?!?!
- Then I tried with a different setup.
SETUP 3 for test
Device 1 = MacBook Air 2020 over WIFI
Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
Device 3 = Apple TV 4 as my second display using AirPlay
Ubiquiti EdgeRouter 4 replaced with an old Apple AirPort Extreme routerAruba AP22 (1 have 3 of these)
Aruba 1930 8 port replaced by a TP-LINK POE switch
Problem with SETUP 3NO PROBLEM AT ALL. No packet loss with device 1 and no AirPlay disconnection between device 1 and 3.....
WHY ?!?!?!?!
- Then I tried with a different setup.
SETUP 4 for testDevice 1 = MacBook Air 2020 over WIFI
Device 2 = Raspberry PI .... just to have access to another OS command prompt over WIFI
Device 3 = Apple TV 4 as my second display using AirPlay
Ubiquiti EdgeRouter 4 replaced with an old Apple AirPort Extreme routerAruba AP22 (1 have 3 of these)
Put back my Aruba 1930 8 port
Problem with SETUP 3NO PROBLEM AT ALL. No packet loss with device 1 and no AirPlay disconnection between device 1 and 3.....
WHY ?!?!?!?!
My EdgeRouter has IPV6
disabled .... never learn IPV6... a guy from old school.... I setup my router with command line... I said I am an old school guy. :-)
My old Apple AirPort Extreme has IPV6
enabled ..... or a clue here!
My MacBook has IPV6
enabledBUT the Raspberry Pi always worked... never had ping issue or packet loss at any time..... no IPV6 at all as well was set on this device.
Remember, all people were complaining when using a MacBook Air or other Apple computer.
So I came up again with the SETUP 1....... the original setup I REALLY WISH TO USE !!!! What did I do that solve my problem? On the MacBook Air, I turn off COMPLETELY protocol IPV6 by using this command :
networksetup -setv6off Wi-Fi
See here for more info :
https://pall.as/ipv6/ipv6-on-mac/
I suspect here IPV6 from the MacBook was bringing network confusion ... using IPV6 momentary instead of IPV4.... when broadcasting over the network or dealing with the Apple TV AirPlay steaming.... for an unknown reason.... I can't explain.... like I said, I never learned or used IPV6 at all..... an old school guy back in the 90'. :-) And because my EdgeRouter does not have IPV6 setup.... no answer from the gateway on IPv6.
I also disabled IPv6 support on the 1930.
So the end of my story..... if you have devices (especially Apple computer) with IPV6 enabled and not have a complete IPV6 network properly setup.... especially setup and full handled with your router..... Just disable it entirely.
So now with SETUP 1.... I have no more, I said NO MORE, issue since IPV6 has been disabled on my MacBook Air 2020 using the command line.End of my story and case closed.... as I am concerned.
Jeff
------------------------------
====
Jeff
------------------------------
Original Message:
Sent: 10-27-2021 01:19 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
I can confirm that removing the 1930 switch entirely from my network has resolved every single network issue I have seen, including Apple TV streaming. Not one hiccup since. I removed the switch yesterday morning, and everything has been perfect since. Aruba is sending me a replacement 1930 switch to try, but I really think it's firmware and not the switch, since it used to work fine and problems started after firmware updates. Not to mention others having the same issue which points to faulty firmware, again since it worked in the past. I can't say for certain the cause on the switch, I can only say that the 1930 switch is 100% the culprit and removing it fixes it all.
Original Message:
Sent: 10-25-2021 10:20 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Yes I am frustrated. I am frustrated that so many manufacturers put out equipment, that has obvious issues. They skimp on solid testing, and put out issues, then expect us to test them for them. This is an attitude that has become so prevalent anymore. I own a 20 person MSP and I get burned out on issues like this. It's cheaper for me to throw the $300 switch away than it is to spend a day performing test after test, documenting, relaying info, etc. Sure I get it for complicated scenarios, troubleshooting SAN performance in densely populated virtual environments, whatever. But for a simple setup of a couple of APs and a switch from a single manufacturer, and you just want basic functionality to work, when you didn't buy off the shelf stuff at Wal-Mart, well it gets old.
Anyways, I am starting to think there are more issues with the switch, and not just with AP22s. I actually don't think there is anything wrong with the AP22s any longer, and I am 99% sure it's the switch alone. Windows PCs on the switch are perfect all the time. Wired Apple TVs (6th gen) of which I have 4 seem to also drop periodically. They drop streams for a 5-60 seconds on occasion when the internet is rock solid. I can't ping out from them so they are harder to troubleshoot. Just like Jeff and Ebakke84, I am about to pull this switch completely from the network to see if it stops that as well. I already resolved all of my wireless issues by not passing the APs through it, so now it's time to move all traffic off of it I guess.
I'll sit on the switch for a bit but if this isn't resolved soon it will have to go.
Original Message:
Sent: 10-24-2021 04:28 PM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
Thanks for your input! It seems I have blue sky ahead soon…. HiHiiiiii!!! 😝😜🎉 . At least. I spent way too much time trying to solve this issue with others who are also searching for a solutions without much feedback from The Manufacturer.
I am not throwing rocks here… I am just explaining a situation where a lot of people are looking for solutions since a while… and we are not having much feedback from people who should be more involved in trouble resolution.
Looking forward to receive my new switch and test everything again.
Regards.
------------------------------
====
Jeff
Original Message:
Sent: 10-24-2021 01:35 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
I have replaced 1930 with fortiswitch, and also tested with cisco CBS250, all good and no more packet loss. (macbook air m1 and apple tv as clients)
aruba, please fix
Original Message:
Sent: 10-24-2021 12:43 PM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
Bad news! The disconnection issue still occurs randomly, even if last Friday, I had no disconnection for all working day.
I am still experiencing packet loss when pinging over wireless from my laptop to any of my AP22's Ethernet port. I noticed that if I am pinging my laptop over wifi from my router, I have no packet loss….. weird….. Weird…..
Any way, I have no other POE switch here to test further without the 1930 at the moment. So I ordered and I will receive tomorrow a new POE switch from a different manufacturer. I will redo all the same tests using the new switch… ping test, AirPlay streaming, etc…without the use of the 1930. Then I will see upon the results I will have what would be the next step…… I am totally fed up to try and try and looking for solution since last summer!!!!!!
Hey Aruba…. What are the results you had in lab??? I am also disappointed that there is no way to disable traffic categorization in AP22…. That bring a lot of information into Aruba servers…. Even if we don't want to. But this is another story.
Jeff
------------------------------
====
Jeff
Original Message:
Sent: 10-24-2021 09:50 AM
From: Unknown User
Subject: Problem AP22 and 2.3.1
I unboxed a 1930-24 today just to look at the local interface, never have and on my box global storm control was off and on individual ports unicast storm control was off while broadcast and multicast were on. Was this the case in your box or did Aruba change the default config overnight so when my box came online and updated unicast storm con is now off on all ports.
There really needs to be the ability to have at least the switches both cloud and locally managed. It would be nice if we could also have full access through the cloud for those times when we don't also install a vpn router.
Now if this was a recent change did they also change already deployed switches? Turn off unicast until they sort out the issue? Would be nice if I could remotely look and see and change if I need.
Original Message:
Sent: 10-23-2021 08:31 AM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
Hi WiFi Guys,
As of now, I cannot, without any doubt, state that AP22 is responsible for what so ever reason into this case. But one thing you brought to my attention is the fact that «maybe» the default value for unicast flow control is too much agressive ??? Might be, might not???
At the moment, I intensively testing my MacBook Air 2020 with my second display using an Apple TV 4 using Airplay protocol..... and let me tell you that it is rock solid now !!! Houray!! I was looking for that solution that gives me a second monitor on my MacBook Air using a M1 processor.
So at the moment, unicast is totally disabled on all ports. I did not mention that I also disabled multicast and broadcast storm control too. Removing such option require to disable the feature globally, then port by port when needed.
The next step would be to enable one feature at a time and take note of the behavior.
It is really a joy. When I was debugging the AirPlay stream on my Macbook by looking at the debug log, when the communication was dropping, logs on my Macbook was telling me that the network connection was abruptly dropping...... Now I understand the network drop was coming FROM the 1930 switch. :-)
A few sad things here, ,you must manage the switch locally, not by the cloud, to have access to all functionalities to be able to do that..... and the another sad thing.... if it is indeed the unicast storm control that is the culprit that stop momentary the traffic here for a wrong reason..... why there is NO information seen in the switch log with an event stating the switch block something ???? No info at all !!!
I never seen errors in any counters... so that makes difficult to debug when you should see some where indication that something is going wrong.
Like I said previously, the next step would be to re enable one by one storm control features and see when the issue will occur again to understand WHERE is coming from. For sure, there is something weird that comes from the 1930 switch..... AP22.... really not sure.
I am still having packet loss below 1% after 10000 ping over wifi when using the 1930 and an AP22. Is it related to the AP22, or only the 1930 again??? Cannot tell, I do not have another switch with POE... sorry. But for sure, there is NO reason why I should have packet loss on an idle network.... I really mean idle network traffic.
Gee.... I am so happy to have finally found a work around..... My home/work setup was not optimal with 1 monitor missing. :-) :-)
I think Aruba support should have a lot of inputs to gather and test from here.
One thing, Apple AirPlay seems to be, like any unicast streaming protocol, looking for performance.... audio and video as a external display.... but there is not reason a 1930 switch cannot handle that properly....... especially on an idle network.
This is my Saturday network report. :-)
Regards.
------------------------------
====
Jeff
Original Message:
Sent: 10-22-2021 07:01 PM
From: Ethan Kozak
Subject: Problem AP22 and 2.3.1
That's some solid troubleshooting Jeff! I wonder if the Unicast Control rate is set too aggressively by default in this firmware update? Strange that it's only AP22s that trigger the issue. Hmm.
------------------------------
Ethan
Original Message:
Sent: 10-22-2021 03:37 PM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
Hi,
I discovered something interesting.
Since my problems initially were all related to a random disconnection between my MacBook Air 2020 when streaming over wifi using AirPlay 2 protocol thru a Apple TV 4 acting as a second external display.
Airplay protocol is indeed an audio+video streaming protocol over IP sending unicast packet to the destination. I decided to completely disable unicast storm control on the 1930. Believe it or not, I am now streaming from my Macbook to my Apple TV 4 without disconnection since the last 4 hours so far. I never got any stable streaming results without disconnection between the 2 devices before.
So far, it seems that it is something related to the default threshold value dealing with unicast storm control. As of now, I will leave unicast storm control disabled for testing purpose until I can confirm that I have no disconnection at all. So far, so good...... 3x AP22 with a 1930 seems so work now.
FL0 : it is indeed related to the use of a 1930 switch. HiHiiiiii!!! 😝😜🎉
To Aruba guys... one more thing...
When trying to keep alive my local browser sessions with my 1930.... default value is 10 minutes before disconnection..... even if I set it at 60 minutes... it will disconnect at 10 minutes....
------------------------------
====
Jeff
Original Message:
Sent: 10-22-2021 07:28 AM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
I still can't explain WHY when pinging over wifi when using an AP22 and a 1930 results with packets loss. Pinging over wired connection, no issue.
Aruba, you should begin in your lab by testing this one first.
And for the sake of this group….. then give to us your results HERE.
Regards.
------------------------------
====
Jeff
Original Message:
Sent: 10-21-2021 12:44 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Oh a couple more:
Let's take a moment to acknowledge that the Aruba switch works perfectly fine without the Aruba APs.
Let's take a moment to acknowledge that the Aruba APs work perfectly fine without the Aruba switch.
Let's take a moment to acknowledge that the problem only happens, when you combine the Aruba APs with the Aruba switch.
That should cover it.
Original Message:
Sent: 10-21-2021 12:38 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Let's take a moment to acknowledge that this is 100% an Aruba issue, that exists only between Aruba APs and Aruba switches.
Let's take a moment to acknowledge that Aruba couldn't make any config changes to their equipment to resolve it.
Let's take a moment to acknowledge that I have already spent countless unpaid hours on this.
Let's take a moment to acknowledge that sure, I first thought it was the APs, but it was still an Aruba issue, as it ended up being the Aruba switch.
Let's take a moment to acknowledge that I tried to avoid issues like this by staying with the same vendor for APs and switches.
Let's take a moment to acknowledge that they haven't even asked me for guidance on internal testing, or even shared any results with me, or even let me know how/when/what they were testing.
Let's take a moment to acknowledge that my time is valuable, and I shouldn't have to waste so much of it on a single switch and 4 APs from the same vendor.
Let's take a moment to acknowledge that others are having the same issue.
Let's take a moment to acknowledge that you putting this back on me and saying I have a bad attitude is off putting. Peace.
Original Message:
Sent: 10-21-2021 10:17 AM
From: Ethan Kozak
Subject: Problem AP22 and 2.3.1
Well, let's take a moment to acknowledge that you were 100% certain that it was the AP22s, and you dragged them through the mud, so to speak. Now it's the switch.
It's understandable that you'd be frustrated, but your constant negative attitude and words are off putting.
Of course you have to do more testing - if they can't replicate the issue in the lab they have to go to the source of the problem. Isn't that pretty standard protocol for Cisco and Ubiquiti too?
------------------------------
Ethan
Original Message:
Sent: 10-21-2021 09:29 AM
From: Unknown User
Subject: Problem AP22 and 2.3.1
I would disagree that Aruba has been working closely with me. Replies are few and far between. Almost all of the testing has been put on me, and I don't have the time to continually test, breaking my wireless in the process. No one even replied to me the results of internal testing or asked me for any input at all in the methodology. In fact, this is the first I've even heard that internal testing even began. It's pretty clear there is an issue with the hardware/firmware on the switch. I can run anything through it (that I have tested) with zero issues, except for Aruba AP22s. I'm losing faith in this product. I have requested an RMA at this point to see what that does as I have nowhere else to go except to sell this switch on ebay, hope it works for the next guy, and move on. To anyone else, stay far away from Aruba switches if you plan on using them with Aruba APs.
Rob
Original Message:
Sent: 10-06-2021 05:13 AM
From: Dhanasekar Panneerselvam
Subject: Problem AP22 and 2.3.1
The Aruba Instant On team has been working closely with the user who has raised the issue and is continuing to work with them.
Despite trying to reproduce the issue, so far we have not seen this in-house.
We want to assure customers and partners that Aruba will continue looking at this to validate this issue.
------------------------------
Aruba Instant On Communications
Original Message:
Sent: 10-01-2021 07:53 AM
From: Oliver Meyer
Subject: Problem AP22 and 2.3.1
Same problem here. It would be nice if someone from the Aruba team would respond to this thread.
------------------------------
Oliver Meyer
Original Message:
Sent: 09-30-2021 03:03 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Sounds like the the D team engineers get this line like with Cisco and the A,B,C engineers deal with the enterprise gear which I get but…. I do have some AP22's on the shelf but not sure about a 1930 switch. I do have a couple operational combo's in the field that I could play with remotely but unfortunately I didn't use a VPN capable router cuz I consider these budget jobs so I just use the ISP's. Now if we had more capability with the web portal, maybe even if the portal allowed a telnet connection we could do things with command line prompts.
I guess I should see what 8 port POE switches I have in stock and plan on a road trip. Fortunately while the two jobs are over an hour away they're within minutes of each other.
Original Message:
Sent: 09-30-2021 02:00 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Not really. They said they were going to lab it up with Engineering but I just got an email today asking me to do even more tests. I don't even have the APs on the switch anymore. At first it sounded promising but I can't get clear information from them. Sounds like they just do enough to kick the can down the road.
Original Message:
Sent: 09-28-2021 05:17 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Has Aruba actually acknowledge this issue yet? Usually a company with let the community know they are aware of the issue and if they were able to replicate on their test bench. At least then we would know there are engineers actually working on this issue. Their silence isn't very assuring that a fix might be in the works and its not very assuring that this product line is getting the attention it deserves or we deserve. Reminds me a lot like Cisco and their SMB line and now long it took to get critical updates for basic functions to work properly.
If I have to pull these switches in order for the AP22 wifi networks to be reliable I will but I won't be using these switches in the future if Aruba can't even keep us informed or even acknowledge the issue and that's its been replicated.
Original Message:
Sent: 09-16-2021 02:18 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Just as an update, I opened a case a couple weeks ago. I have been testing the living hell out of this. Aruba support felt it might be the 1930 switch causing an issue, so they moved it to local config mode, and made changes to LLDP. However, that didn't fix anything. They have reviewed everything and can't find an issue. I just got an update that they are escalating this to engineering.
So we aren't sure yet if it's the 1930 switch, or the APs. All APs are connected directly to this 1930 switch. Wired clients on the 1930 switch have no issues ever and can ping the local gateway all night long and not drop a single packet. Wireless clients connected to AP22s, that are subsequently connected to the 1930 switch have this issue where they may drop as many as 80 pings in a row to the local gateway, often several times per hour, all while also still being able to ping a local server. It's really bizarre how they are denied access to only the local gateway, and subsequently the internet, but not other hosts. Again, wired clients on the 1930 switch never ever drop to the local gateway.
Does anyone else have AP22s connected to a 1930 switch? If so, could you do some extended pings to your local firewall and see if the issue can be reproduced? I think I might take one of my AP22s, and connect it separately to a different switch to see if it's the 1930 causing it, or the APs.
Original Message:
Sent: 09-01-2021 05:16 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
That sounds identical to what I am dealing with. It wasn't always like this though, a recent firmware update caused it. I am currently on the phone with Aruba creating a case. For me, the lengthy drops could be anywhere from 15 to 60 seconds long, and can happen anywhere from 1-4 times per hour. I haven't been able to find a pattern to it. But I know it's not specific to the AP as I have 4 of them and they all have the issue.
I wouldn't drop them yet. I think this is just a bug for now. I came from Ubiquiti to these and these were night and day better than Ubiquiti a first, but now the firmware updates broke them a bit.
Original Message:
Sent: 09-01-2021 04:44 PM
From: Jean-François Ménard
Subject: Problem AP22 and 2.3.1
Hi,
I am dealing with a similar behavior here. I have 3x AP22 running, with an Aruba 1930 and a Ubiquiti EdgeRouter 4.... I am using my MacBook Air 2020 to stream using AirPlay2 over an Apple TV 4 HD. No matter what I do, it is always have dropping at an intermittent amount of time... 5 min, 1 hour.... no specific time.
This is really frustrating. I am using this Airplay2 streaming as my 2nd monitor, and the streaming is dropping quite many times per day. I carefully followed Apple advices on how to choose right WiFi settings: channels, bandwidth, etc.
Not much luck so far.
I re-tested with my old all-in one wifi-router from Synology RT-2600AC alone... nothing else.... everything is stable, no issue at all. The reason I changed everything here is to let me have a better WiFi coverage.
I am seriously considering dropping all my Aruba gears for another vendor.
Jeff
------------------------------
====
Jeff | VA2SS
Original Message:
Sent: 09-01-2021 10:47 AM
From: Jessica Mitchell
Subject: Problem AP22 and 2.3.1
Hi,
We want to assist you in getting this issue resolved. We request you create a support ticket with our Aruba Instant On support team and include setup, all client issues seen, and any additional concerns relating to the issue. This will help our team assist you in narrowing down and resolving the issue. You can reach us at: https://www.arubainstanton.com/contact-support/
------------------------------
Aruba Instant On Communications
Original Message:
Sent: 07-12-2021 04:38 PM
From: Unknown User
Subject: Problem AP22 and 2.3.1
Having major problems with 2.3.1. Macbooks seem to have an issue where they are connected just fine, but can't contact their gateway on occasion. If I get 2 pings going, one to a local server, and one out to the internet, the one to the local server doesn't drop. However the one out to the internet will drop for maybe 20-30 times in a row and the magically come back.
These are 3 AP22s connects to a 1930 switch, with a SonicWall as a gateway. I have confirmed the SonicWall isn't blocking anything. The same Macbooks I can connect to the ISP provided wifi with 0 issues. So it's nothing like that.
I don't know, these AP22s have been solid, and now they are just flaking out everywhere even though they are up to date and I've rebooted them. I'm always at max signal on the macbooks. Real worlds transfer speeds are above 300mbps. They just suddenly can't communicate with the gateway, yet an iphone right next to it, connected to the same SSID, is sitting there running speed tests without any drops, all while the macbook can't get to the gateway.
I've turned off Wifi6. Not sure what else to check.