Intermittent Enterprise Outages and WorkFlows Slowdowns

PrintPrintDownload PDFDownload PDF

Members are reporting slowness with Symphony Workflows and connecting to Enterprise, with occasional server unavailable message in Enterprise. We are working with SirsiDynix on identifying the source of the problem. When this occurs we are seeing server load averages increase. 

Reported:

  • 12/26 15:24 PM
  • 12/27 12:12 PM
  • 1/2 12PM (approx.)
  • 1/3
  • 1/4
  • 1/7

We have disabled the Novelist widget in Enterprise as this appears to be what is making excessive Web Services calls to our Symphony ILS server.  We will be working with SirsiDynix and EBSCO to review the problem. This service will remain deactivated until we have determined it can operate without impacting Symphony server load.

Monday, January 28th Update: the library holdings on the search results page was re-enabled. We are monitoring performance of the widget code.known issue library holdings

Library holdings do display on the item details page. Follow the title link to see the item level holdings.

Tracking Updates: 
Issue Updates
Date/TimeNote
December 27, 2018 - 2:00pm
With Novelist disabled we continue to experience Enterprise outages and web service calls. We have disabled Enterprise web service calls while SirsiDynix continues investigating the problem. As Enterprise is impacting Symphony Workflows, we have elected to keep Enterprise running in a degraded state so that it does not impact Workflows.
December 27, 2018 - 4:00pm
Web Services has been opened on Enterprise allowing patrons to log in to My Account. We have disabled Item Availability on search results pages but it should continue to display on individual item detail pages.
January 2, 2019 - 2:00pm
We've continued to observe these load spikes this afternoon and have determined a connection to Enterprise and Web Services. That being the case, we've temporarily suspended Web Services connectivity to Enterprise, meaning Item Availability, Holds, and My Account features will be unavailable as we troubleshoot.
January 2, 2019 - 5:00pm
A quick update to our intermittent Enterprise outages and Symphony/WorkFlows slowdown issues. We’re continuing to see influxes of Web Services calls to the Symphony server, which have resulted in intermittent Enterprise outages and occasional performance spikes on the Symphony/WorkFlows server. These issues first arose December 26th and 27th, and had subsided until this afternoon. To troubleshoot further, we temporarily disabled Web Services access on Enterprise for several hours today, which impacted My Account logins, Item Availability, and Holds placement. Those functions have since been restored and are expected to remain online throughout the evening. Fortunately, this troubleshooting has gotten us much closer to pinpointing the source of the issue and we will continue to investigate and monitor the situation throughout the week. This ongoing investigation and intermittent outages have been noted to patrons as a banner on all Enterprise profiles. Additionally, you can track new updates to the case via the support site posting. In the meantime, we expect that significant Symphony/WorkFlows slowdowns should largely have subsided, so the majority of the continued impact will be on Enterprise. We thank you for your continued patience as we work to resolve this issue.
January 3, 2019 - 8:45am
We have re-enabled library holdings on the search results pages and Novelist on the item detail pages. We have ruled out these features as the source of the increased Web Services calls.
January 3, 2019 - 1:15pm
We are still seeing influxes of Web Services calls to Symphony, and we have to periodically disable Web Services access to Enterprise. You may see that My Account logins, Item Availability, and Holds are unavailable throughout the day. We have also disabled the local library holdings display again to reduce the number of Web Services calls.
January 3, 2019 - 3:45pm
We've temporarily disabled the holdings display on the item detail and search results, and so far we're seeing improvements to the load averages. In addition we've temporarily disabled the ability to perform a "zero search" ( a search with no search terms). Thank you again for your patience as we troubleshoot this issue. Access to My Account is restored for the moment.
January 4, 2019 - 12:15pm
At the moment we have left the holdings displays disabled, and we plan to leave them disabled through the weekend. With the holdings disabled, Enterprise and Workflows are stable and patrons are able to log into their accounts and use hold functions. Our priority is to make sure patrons can access the most important services while we troubleshoot these issues and keep Workflows up and running for staff. We are waiting for an update from SirsiDynix just a little later today, but we do not have an ETA as to when these issues will be resolved. You may see holdings displays appear or disappear throughout the day as we troubleshoot. Thank you for your patience and we will continue to update this issue.
January 4, 2019 - 4:45pm
We enabled the holdings display for the item detail this afternoon, and Enterprise remained stable. We're leaving it enabled through the weekend unless we see load increases again. The holdings display for search results remains disabled. We tested enabling it this afternoon shortly after 1pm and saw the load increases return. We're in continuing talks with SirsiDynix, and we will be monitoring the issues over the weekend.
January 7, 2019 - 4:00pm
Today we completed another call with SirsiDynix Support and Engineers to review the issue and have narrowed the source of the problem. We will leave the search results local holdings display disabled while this work continues. However, these holdings are visible within the Item Display detail page. We recognize the inconvenience to your library patrons and appreciate your patience.
January 22, 2019 - 2:00pm
We are currently in a system freeze period while bibliographic, patron, and circulation data is being loaded for Green Hills. The Green Hills go-live date is January 29th. When this completes, our priority is getting Enteprise search results holdings restored. We apologize for the delay in resolving this issue.
February 1, 2019 - 11:15am
Novelist Select displays have been restored.
Status: