Yet another one of those interesting discoveries that took most of my afternoon to figure out.
What I was trying this time was synthetic test cmdlets available to test different workloads in Lync Server 2010. I was testing IM conferencing. I had two users in this scenarios homed on different pools. I was trying to make sure everything is configured correctly and the users in this test can initiate an IM conference and invite other users.
First I tried using Test-CsGroupIM and Test-CsAVConference cmdlets against user’s home pool, using the following command syntax:
Test-CsGroupIM –SenderSipAddress userA@contoso.com –ReceiverSipAddress userB@contoso.com –TargetFqdn se01.contoso.com -Verbose
Test-CsAVConference –SenderSipAddress userA@contoso.com –ReceiverSipAddress userB@contoso.com –TargetFqdn se01.contoso.com –Verbose
Both came back resulting in failure:
TargetFqdn : se01.contoso.com
Result : Failure
Latency : 00:00:00.8463930
Error : This operation has timed out.
Diagnosis :
The operation was timing out, but it didn’t make sense. My configuration was correct, I had windows firewall enabled during installation of Lync Server and I could see all the CS rules in firewall configuration. It shouldn’t be a firewall issue. So, I tried to manually initiate a conference using Lync 2010 client on a client machine. This test was successful. I had 3 users in IM conference and it was initiated by userA homed on se01.contoso.com. Another pool in question was se02.contoso.com and was configured properly from what I can tell. Now I was getting into the weeds.
I looked at the results from test cmdlets mentioned above and I noticed the following:
'Register' activity completed in '0.2563212' secs.
'Register' activity started.
Sending Registration request:
Target Fqdn = se01.contoso.com
User Sip Address = sip:userB@contoso.com
Registrar Port = 5061.
Auth Type 'Trusted' is selected.
Registration Request hit against Unknown
'Register' activity completed in '0.4517277' secs.
'CreateConference' activity started.
'CreateConference' activity completed in '0.0150415' secs.
'JoinConference' activity started.
'JoinConference' activity completed in '0.4045165' secs.
'JoinConference' activity started.
'JoinConference' activity completed in '0.4232243' secs.
'SendGroupIM' activity started.
An exception 'This operation has timed out.' occurred during Workflow
Everything seems to work including CreateConference and JoinConference, until SendGroupIM activity step. That’s when it times out. I have to figure out what is it timing out on as Lync client is working as expected without experience the timeout issue. So I started logging tool and enabled S4 and SIPStack components. After running the test again, which failed as expected, I looked at snooper. What I found, I can say, wasn’t a surprise. Or was it?
The figure above shows failed to complete outbound connection. What was it failing on? Looking at the details of that message in snooper, I found the following (showing only relevant info) :
…
Peer-IP: 192.168.1.15:49950
Peer-FQDN: se02.contoso.com
…
Transport: M-TLS
Result-Code: 0x8007274c WSAETIMEDOUT
Data: fqdn="se02.contoso.com";tls-target="se02.contoso.com";peer-type="InternalServer";winsock-code="10060"
$$end_record
What I noticed was that it was timing out when trying to reach other SE on port 49950. If you remember the RDP/SRTP workload ports, they are 49152-65535. And this port is one of them. When trying to initiate GroupIM, it was failing on that port! While windows firewall was configured by Lync bootstrapper as expected, obviously, that wasn’t helping synthetic test cmdlets! Remember, it wasn’t affecting actual user testing, it was just these tests that were failing. I don’t think I can explain why. I believe it has something to do with built-in port security where ports are only opened by Lync for connections authorized. Is Synthetic test trying something funny here? Is authentication an issue? I am afraid, I don’t have clear answer for you on this one, but I do have solution!
So, what did I do to fix the testing issue? I simply created a rule in firewall that allowed TCP 49152-65535 port range instead of relying on Lync server to decide for me using built-in CS rules. As soon as I created that rule on SEs, the synthetic test cmdlets returned success! Bhargav – 2, Firewall – 0. J
This sounds like the issue many oterhs have seen with the CS* RBAC groups versus the RTC* groups. In my understanding, the CS* RBAC groups only grant permissions to run Lync cmdlets remotely (i.e., via Powershell remoting), whereas only the RTC* groups grant permission to run cmdlets locally on Lync servers. Seems this is the behaviour both in Lync Server 2010 and 2013.