We have an update for you after our meeting with Vista Hosting, BB Support and others. In general, BB has been looking at our server performance and what may be contributing to the erratic issues and general slowness that we have seen recently. I will list them here and the actions for each. This is not in the order of importance, as it is hard to quantify how intensive something degrades performance but the order that BB addressed them:

1. One section in one college had 10's of thousands of enrollments in a general purpose student orientation course. When the students in this section would get particularly active, it was tying up the threads. Hosting was seeing page requests in this section logging 15 - 30 seconds and at an extreme 15 minutes. As more and more users were trying to access course tools, the software was getting further and further behind in processing these requests. They had to restart the node to get it cleared out. The normal range for section enrollment that Vista has seen work is up to 2000. This is not a hard and fast rule, but CCCOnline experienced some issues on Campus Edition in the past when we had 5,000 enrollments in our Student Orientation class. We had to do our orientation outside of WebCT and created web pages for that.

Solution -- This issue is currently being addressed by the college.

2. The scaling of our hardware to the user level needs to be reviewed. We do upload all users from Banner for the term for each college. Not all of these are active users and would affect the performance of our servers, but we really don't know whether our five node system is scaled for the level of users we actually have. We do know that we are increasing our users much more quickly than originally planned.

Solution -- Hosting will be doing some analysis in the next week to see whether there is a capacity issue. The analysis would be based on who has actually accessed a section in the last 30 days. If extra nodes are needed, these will be deployed by BB hosting immediately.

3. Erratic Vista performance, various tool and access issues for the last week.

Solution -- Application Pak 1, Software Pak 1 upgrade recommended for stability and bug fixes. This application pack came out this summer and Hosting recommended we wait until their was a bug fix for this, as they were aware of potential issues. The fix has now been incorporated into the Application Pac, which also upgrades several tools and such. The only known bug in this upgrade is the inability for faculty to upload a grade book or .CSV file into Vista. The application pak has been used for other large institutions both Vista hosted and large self-hosted configurations. Vista Hosting has scheduled many such deployments for the next several weeks. They have not seen any adverse affects and believe the risks are minimal. That said, we are making arrangements to check the application once it is deployed early Saturday morning. With an 8 hour window, we have time to fall back to our original setup, if need be. I will send out further details for those of you that would be willing to check your institution early Saturday morning for any issues.

4. Our enrollments and user files were running past a deadline we had set to not cross. We have not been able to monitor this because the imports have been emailed directly to the college vista administrator and not to us at CCCOnline. Because of the extensive missing users issues we had from Banner for most of September and into October, we were sending full enrollment imports at 10 AM and 11 PM each day, but we were trying to avoid mid-day through the evening when most users were in Vista.

Solution -- We have a new improved script starting in the next day and our imports to Vista beginning last night and today are only differentials or changes in enrollments, rather then full for each college, which cuts dramatically the processing time for Vista. This takes some of the load off of the server.

Finally, we want to be more proactive on maintenance. We would like to schedule a maintenance window for each weekend during which time BB hosting can check the servers and do a re-start. The re-start will help to clear out threads and clean things up each week. We will contact each of the colleges next week to talk about a time to do this each weekend. Vista (BB) Hosting has also indicated that they will help us clean up the various databases and eliminate the extraneous users that are currently included in those databases due to course migration from BlackBoard, WebCT and Vista4 Hosting services.

In summary -- we are working on a three pronged approach to the problems we have been having recently: changing a few business practices, adding hardware, applying software fixes. By Saturday morning we hope you will all see significant improvements.

David Chatham
CCCOnline