Hi Hilary,
Thank you so much for taking a peek at this! All of our server patches are up-to-date.
So after some digging I was able to narrow down info from the logs. The log is very long, but I thought these first 4 lines might provide a clue. On a side note, all of these lines started with "SharePoint Server Distributed Cache Unexpected". The first line says "Ensure that security permission has been granted for this client account" which might be my best clue. Does this ring a bell? I thought I had Full Control everywhere, but could there be somewhere that you could think of that I am missing permissions? ....or could it be something else?
LINE 1:
Unexpected Exception in SPDistributedCachePointerWrapper::InitializeDataCacheFactory for usage 'DistributedActivityFeedLMTCache' - Exception 'Microsoft.ApplicationServer.Caching.DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.) ---> System.ServiceModel.EndpointNotFoundExcepti...
LINE2:
...on: Could not connect to net.tcp://xxxsps2:2xxxx/. The connection attempt lasted for a time span of 00:00:00. TCP error code 10061: No connection could be made because the target machine actively refused it 10.1.xx.46:2xxx3. ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.1.xx.46:2xxx3 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout) --- End of inner exception stack trace --- Server stack trace: at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout) at System.Ser...
LINE 3:
...viceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout) at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout) at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ApplicationServer.Caching.CacheResolverChannel.Open(TimeSpan timeout) at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Object[]& outArgs) at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message r...
LINE 4:
...eqMsg, Boolean bProxyCase) at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) at Microsoft.ApplicationServer.Caching.CacheResolverChannel.OpenDelegate.EndInvoke(IAsyncResult result) at Microsoft.ApplicationServer.Caching.ChannelContainer.Opened(IAsyncResult ar) --- End of inner exception stack trace --- at Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody, RequestBody reqBody) at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCacheProperties(RequestBody request, IClientChannel channel) at Microsoft.ApplicationServer.Caching.DataCacheFactory.GetCache(String cacheName) at Microsoft.Office.Server.DistributedCaching.SPDistributedCachePointerWrapper.InitializeDataCacheFactory()'.
Thank you!