Hey Everyone- This note is going up on Blackboard Vista today:
Attention Students and Faculty: Temporary Service OutagesThe splash page will be up during the brief downtimes and will include information about the duration of the down time.
The Vista System will be taken down and brought back up daily at 2 PM, 8 PM and 4 AM for approximately ten minutes.
BlackBoard has identified the core issues with the recent latency in the performance. The reboot will be scheduled as a part of a short-term solution until we can schedule a software upgrade.
We apologize for this inconvenience. Please schedule your online activities around these daily system outages.
What the cluster reboot does is reset all of the connections between the application servers (we have 8) and the database. Each server has 100 (rising to 125 later today) connections available. Apparently these are not being released properly once the current request has been processed. Re-booting the entire cluster (all 8 application servers and the database) resets these connections (releases them really) and allows performance to return to normal levels.
The root cause of the bottleneck is problems with the way Web Logic is communicating with the database server. The upgrade to application pack 2 includes an upgrade from Web Logic version 8 to version 9, which does include general fixes for this sort of problem. We will probably try to push that upgrade forward from the currently scheduled weekend (September 22-26), but have to complete some additional testing first. As soon as I have more solid information I will pass it on to you.
Thanks for all of the hard work! Students seem to be in their classes and happy with them. We are receiving very few support calls at this time and the support desk (Deb Michel) has been able to keep up with the calls and emails all week.