*Online, Continued Monitoring* SWAN Server Migration Complete, SIP2 Up

PrintPrintDownload PDFDownload PDF
Added by Steven Schlewitt (12/05/2018 - 14:23) , last updated by Aaron Skog (01/18/2019 - 10:05)

--Update at 11:00am on 1/16--

The SWAN IT team has been closely monitoring the Symphony/WorkFlows server throughout the morning, and have yet to encounter any instability that has impacted the server over the last two days. At this point, all services have been fully restored, including SIP2, Web Services, harvest agents, and acquisitions. If you continue to experience issues with any of these services or have been unable to access the SWAN server, please let us know by emailing help@swanlibraries.net.

For the remainder of the week, we will continue to closely monitor the system load and fine-tune application and server settings. We'll send additional SWANcoms and support site postings should any news become available. 

Thank you for your continued patience throughout the server migration process and the last two days of troubleshooting.

 

--Update at 3:40pm on 1/15--

The Symphony WorkFlows server appears to be stable after our recent round of troubleshooting. That being the case, we’ve reenabled all SIP2 and Web Services functions to increase the overall system load, so all library SIP2 ports are now open. Additionaly, all WorkFlows services are currently functioning as expected.

We’ve also lifted the freeze on acquisitions. Please feel free to begin processing acquisitions at any time. To retain the correct report configuration, you’ll need to unsuspend the acquisitions reports relating to your library. Please refer to the support site documentation for additional information on how to unsuspend or reschedule those reports.

If you are still having difficulty connecting to the WorkFlows server or have encountered any other system anomalies, please open a new ticket by emailing help@swanlibraries.net.

Thank you for your continued support and please continue to watch SWANcom for updates.

 

--Update at 2:45pm on 1/15--

The Symphony/WorkFlows server is back online and available for logins. Please login normally at your earliest convenience.

Once again, we are continuing to troubleshoot, so we may experience additional outages our slowness during the afternoon.

We’ll send additional updates via SWANcom and support site post as information becomes available.

 

--Update at 1:10pm on 1/15--

We’ve planned an outage of the Symphony/WorkFlows server at 2pm today to further test a system configuration and correct an issue with print label templates. During that time, WorkFlows, SIP, and Enterprise Web Services functions will be inaccessible.

Please be prepared to use WorkFlows in Offline Mode beginning at 2pm until further notice.

 

--Update at 12pm on 1/15--

The Symphony/WorkFlows server is back online and available for logins. Please login normally at your earliest convenience.

We continue to troubleshoot issues relating to SIP and the system load, so we may experience additional outages our slowness during the afternoon.

We’ll send additional updates via SWANcom and support site post as information becomes available.

 

--Update at 10:15am on 1/15--

We’re currently experiencing an outage of the Symphony server. Please proceed to use WorkFlows in Offline Mode as we continue to troubleshoot.

We sincerely apologize for the inconvenience and continue to work to find a resolution to these ongoing issues. We appreciate your continued patience.

 

--Update at 5:42pm on 1/14--

End of Day Update: I wanted to update everyone on the status of the new server supporting Symphony. The SWAN IT has been updating the I wanted to update everyone on the status of the new server supporting Symphony. The SWAN IT has been updating the support website throughout the day, so make sure to keep this URL open and refreshed throughout the day tomorrow.

The good news

This server is well-provisioned and responding in the way we expected. Libraries will be happy with the difference once we have stabilized our configuration. We have positioned ourselves for Symphony to be stable for the rest of the evening. Start your Tuesday as normal, and watch for updates via SWANcom and the SWAN support website.

The bad news

Throughout the day today, SWAN’s Symphony server experienced a spike in server load which caused WorkFlows connections to drop. We have identified a probable cause related to the SIP2 service.

The SIP2 traffic SWAN supports for libraries is supporting well over 100 connections. These are connections to self-checks, automated material handlers, PC Reservation, MyPC, etc.

As we are examining the SIP configuration, we are randomizing the reactivation of SIP connections for each library.

Acquisitions Symphony Reports remain disabled today, but we are looking towards allowing these reports to be run after we regroup in the morning.

Aaron Skog​, SWAN Executive Director

--Update at 1:47pm on 1/14--

We’ve again disabled SIP2 connectivity as we continue to troubleshoot. If possible, please leave your SIP devices connected to the server (do not power them off), and we may need to review that traffic in our logs.

 

--Update at 1:06pm on 1/14--

The Symphony server is again stable and we’re running additional monitoring processes to further troubleshoot.

Please feel free to login to WorkFlows again now. We’ve also reenabled SIP connectivity. We may see additional spikes this afternoon, so please be prepared to use Offline Mode should you get disconnected abruptly.

Thank you for your continued patience as we work through the issue.

 

--Update at 12pm on 1/14--

We’re currently seeing another load spike on the server. Please proceed to use WorkFlows in Offline Mode as we continue to troubleshoot.

We sincerely apologize for the inconvenience and are working with SirsiDynix engineers to find a resolution.

 

--Update at 9:53 am on 1/14---

We believe to have pinpointed the issue that caused our early morning load spike. We’ve reopened connections to WorkFlows, but continue to troubleshoot issues with the SIP2 service.

Please login to WorkFlows as usual and await further updates regarding the SIP2 service.

 

--Update at 9am on 1/14---

We are currently experiencing some high load issues with the Symphony server.  Please use offline mode while we investigate the problem.  We will send an update when we have more information and when it is safe to log in normally.

 

---Update at 5am on 1/14---

We’ve successfully completed our server infrastructure migration to the cloud! The new SWAN Symphony server in the Microsoft Azure Cloud has finished the regularly scheduled overnight maintenance reports and is now available for use.

A few things to note:

  • Please login to WorkFlows and perform all functions as usual. No further downtime is expected today. Additionally, we will not need to use Offline this morning.
  • Acquisitions departments – we’ll be performing some additional testing later this morning before reenabling acquisitions reports. Please watch for an update later this morning.
  • Our report schedule is currently on-track, with most reports running as scheduled.
  • All regularly scheduled reports will run as expected today. This includes autorenewals, patron notices, and clean holds shelf.
  • BLUECloud Analytics statistics may be delayed by one day. We expect this will be caught up tomorrow (Tuesday) morning.
  • All Enterprise, payment, and authentication services are operating as expected, with the exception of item availability in the search display due to ongoing troubleshooting.
  • The new server uses the hostname api.in.swanlibraries.net (same as our previous server). As long as your WorkFlows client is pointing to this hostname, you’ll login to the server as usual.
  • No changes are needed to SIP2 devices or software referring to the current hostname.

We’ll send additional SWANcoms and news posts if any substantial issues arise, so please continue to watch throughout the day. Additionally, if you encounter any issues or oddities, please send them to help@swanlibraries.net so that we can investigate further.

Thank you for your patience throughout the migration process.

 

---Update at 6pm on 1/13---

The SWAN Symphony server migration is underway! We’ll send an update via SWANcom and support site post when our overnight work has completed, likely early tomorrow morning.

As a reminder, please be prepared to start the morning with WorkFlows in Offline Mode unless we give the all-clear prior.

Thank you for your continued patience.

 

---Original Posting---

SWAN Member Library Staff:

We’re at the last milestone of the SWAN server infrastructure migration project! On Sunday, January 13th at 6pm, SWAN IT will work with SirsiDynix engineers to migrate our server data to the new server infrastructure host in the Microsoft Azure Cloud. This migration could take the entire evening into Monday morning, so please check SWANcom and the SWAN support site first thing on Monday morning prior to your initial login.

Unlike the past two milestones in this project, no additional work will be required on the library’s side to accommodate this change. SWAN IT staff are currently wrapping up the coordination of VPN changes with the Network Self-Maintainer libraries, but the Symphony WorkFlows server will continue to use the hostname, api.in.swanlibraries.net and will continue to be accessible exclusively over the SWAN VPN. Some changes to the traffic routing on your library network may be needed – an upcoming benchmark of our new environment will help to determine if this is at all necessary.

 

The Important Details:

What? Symphony server Cloud migration

When? Begins at 6pm. on Sunday, January 13th concluding the morning of Monday, January 14th. (Migration is scheduled after library hours)

Who is Impacted? Primarily overnight/online patrons and potentially early morning library staff

  • Circulation/Reference Staff: Potentially use WorkFlows in Offline mode on Monday, January 14th unless the all-clear is given prior to open.
  • Cataloging/Tech Services Staff: Observe the Acquisitions freeze from Friday, January 11th until notified via SWANcom (expectedMonday, January 14th).
  • Information Technology Staff: If you are a Self-Maintenance VPN library, Rudy Host will be reaching out to you (if he hasn’t already) to configure a new VPN. Additionally, your library may need to adjust network traffic routes if any changes have been made internally in your library (not present on your Sonicwall). Otherwise for libraries with Sonicwalls managed by SWAN IT, the necessary changes should already be in-place.
  • Patrons: Will be unable to access My Account, Item Availability, Payments, and Holds functions in Enterprise during the migration.

 

Service Availability During the Migration (after 6pm on Sunday the 13th)

  • Symphony/WorkFlows Server
    • No access to SIP2 or Web Services 3rd Party Vendor Integrations (databases, electronic media)
    • No access to SIP2 Hardware/Software Integration (AMH, self-check, PC reservation, print release)
    • No access to WorkFlows
  • Enterprise
    • Full access to current item records and searching (cached data)
    • No access to My Account features
    • No access to place holds
    • No access to commerce/payment functions
    • No access to item availability
  • BLUEcloud Applications
    • Full access to BLUEcloud Analytics
    • Full access to BCA report functions
    • Statistics from November 7th, 8th will not be accessible until later in the week
    • BCA Reports will be generated and emailed as scheduled
  • No access to Bookmyne
  • No access to MobileCirc

 

Quick Notes…

  • Migration starts at 6pm on Sunday, January 13th and extends into the early morning of Monday, January 14th.
    • SWAN staff will send a SWANcom in the morning of January 14th to give the “all clear” to login and begin using the migrated server.
  • This migration will require a FREEZE of all Acquisitions services on Friday, January 11th at 12pm, that extends through to the completion of the migration.
    • Acquisitions performed during the freeze period will potentially cause order issues.
  • No WorkFlows client updates or hostname changes are necessary to accommodate this change.
    • SIP2-based PCReservation libraries should verify that their server is referencing the hostname and not the IP address. This feature is available in PCReservation 4.5+
  • Monday, January 14th’s morning report schedule WILL experience delays due to the shift of overnight maintenance.
    • Please avoid scheduling unnecessary reports until the schedule has caught up.
  • BLUECloud Analytics statistics will likely be one day off. These will catch up on Tuesday, Jan 15th.
  • Library staff may need to start Monday, January 14th using Offline mode if the migration takes us into the morning.
  • *NEW: We strongly recommend having your IT staff or contractors available and on-site on Monday the 14th as some libraries may need internal network/routing changes that cannot be performed by SWAN IT staff.
  • *NEW: SWAN will be implementing a new volume email service this week (prior to the migration) to help us better control and monitor email notification success from the new environment.
    • This new service should not require any changes by the library or patron, but we’ll issue further notice if email spam rules or anything of the sort will need to be modified to accommodate.
  • *NEW: Access to digital materials through services such as Overdrive, Axis360, and Hoopla will be unavailable during the migration process. Materials checked out this weekend prior to the 6pm migration may be cached and remain available with select digital services, however.

 

Server Environment Test Announcement

Watch SWANcom later this week for an announcement and details of a test of the new server environment. The test will take place on Thursday, December 13th between 10am-12pm and will require each library to perform a few simple WorkFlows functions in the new server environment and submit a FormSite survey.

 

Questions or comments? Please contact SWAN Support

If you have any questions, comments, or run into any difficulties on the day of the migration, please email help@swanlibraries.net to create a ticket in the SWAN support system or contact SWAN Support directly by calling 844-SWANLIB (844-792-6542) and dial 0 for immediate assistance.

 

--
SWAN Library Services
800 Quail Ridge Drive - Westmont, IL 60559
Email: help@swanlibraries.net
Phone: 844-SWANLIB (844-792-6542)
https://support.swanlibraries.net/