BB Vista is back up

Blackboard completed their testing and upgrading of Vista last night near midnight. Our testing since has not revealed any new problems since that time. However, we will continue to monitor the system as the day progresses.

As a result of the problems yesterday, your students may have had difficult attaching documents, posting announcements or sending class email. Their ability to login was impacted, especially yesterday afternoon. Please take all that under consideration as you and your students complete these last two weeks of classes.


Thanks for all of your patience!

Lisa Cheney-Steen

Weekend Downtime

Hi Everyone-

Please don't forget this weekend's downtime for Blackboard Vista. We will be going down at 1:00 pm on Saturday, April 5th and coming back up again by 1:00 pm on Sunday the 6th. This downtime will be used to install SP3 and a hot fix. I don't think you will notice much on your side, but there are significant changes on the back-end.

Thank you-
Lisa

Downtime

Good Morning Everyone-

We are back up again this morning as of around 6:00. That would put last night's downtime at close to 12 hours. I don't have an update yet on why we were down, but I'll pass on information as I have it. We do appear to be stable at this time.

Lisa

Vista Update

Here's the latest news in a set of emails from John at BB to David:

The problems today are related to the build-up of files that haven't been removed in garbage collection (GC). I (David) am going to paste in John's (from BB) explanations below:

When you e-mailed Lee about having trouble uploading files I went in and added some indexes to the CMS tables and reported how much GC had to do to you. That's why it was slow. Once we reboot the system tonight GC will start cleaning up those 16 million content items in the CMS tables.

Right now GC has been cleaning up old student records and sections but not content items. I think this is why uploads have been so slow.

-John

I just finished adding the tracking job indexes to fix the tracking data.

That job will run Monday Wednesday and Friday starting at 5a.m. Please let me know if you see anymore Unknown Users after tomorrow evening.

I also added the indexes for Garbage Collection. That job will run Tuesday Thursday Saturday and Sunday starting at 10p.m. and ending 8 hours later.

Finally I added indexes to the CMS tables. Uploads are now blazingly fast from my connection here at home and my neighbors DSL connection over wireless-b.

-John

John asked me earlier if we would ask instructors to delete their section backups once they have downloaded them. We need to put into our training and instructions to faculty about not leaving heavy files, like backups, in the MyFile area. As Vista administrators, we also need to delete backups – now would be a good time – so that the database is free of unnecessary data. The GC hasn’t worked, but after tonight we should be on the road to cleaning this up.

Thanks for your patience, and let’s see how the system works for the next couple of days.

David

I will get LisaMarie to write up directions for deleting back-ups you may have made, then if you would take care of that we could begin the semester reasonably clean.

Many thanks!
Lisa

Vista Update

Well folks it doesn't look like BB Vista is really working correctly right now. I don't recommend you do any work in your classes quite yet.

For those of you who are still waiting for your spring shells, David can't dupe.

And last, enrollment is up 24% for spring 1 and 34% for spring 2 when compared to this day last year.

Lisa

BB Vista Database Issues

Hi Everyone-

We are stil on the new databse hardware this morning, but have rolled back to Oracle version 9i from 10g. Several people around the system are testing this set-up as I type. If it appears to be both functional and stable we will settle here for awhile before we try to go to 10g.

On the theory that it's good to have options we also have two open-source learning management systems set up now for testing -- Moodle and Sakai. We don't have any formal plans around moving towards another LMS, but we are exploring options. (Moodle information is at moodle.org; Sakai is found at www.sakaiproject.org.) We may want to run a test course in each of those LMS's or we could offer a training course in each if enough of you are interested. Talk with your chair if you are interested in either project.

Lisa

Vista Update

BB ASP has asked us for the next 12 – 18 hours to try to rectify the Oracle 10 G issues in Vista. They will be applying a patch and rebuilding some indexes overnight. If this is not a successful operation, as far as remedying the issues, they may recommend a roll-back to Oracle 9i for the short-term until the issue is fixed in 10 G. Because of the need to be up and running for OJC on January 7, we have little time to troubleshoot the problem and must find a stable environment for the semester.

A splash page has been put up on the site, directing users to a brief explanation of the situation. I expect to have an update on the situation tomorrow morning sometime. There will be a decision point tomorrow whether the overnight work is fixing the problem or whether we need to roll-back, necessitating further downtime.

(Side-note from Rhonda and Lisa: We would be going back to the old version of Oracle, but staying on the new hardware.)

David Chatham
Dean of Academic Technology
CCCOnline

Database Crash

Hi Everyone-

The BB Vista database has crashed. Please don't try to do anything on Vista until further notice.

At this point all we know is that it is probably as Oracle problem. Oracle has been called in and is working on the problem.

We will let you know as soon as we have more information.

Sigh.

Lisa

Remember that the Blackboard Vista servers will be down for planned maintenance today through Sunday while we move to a newer, larger, faster, database server. We expect to be back up Sunday morning, although the downtime is not officially scheduled to end until Monday morning at midnight.

Hope your break is going well!

regards,
Lisa

Downtime over Christmas Break

I want to wish everyone a happy happy Christmas break!

CCCOnline offices will be closing Friday the 21st and not re-opening until Wednesday, January 2nd. We are making some hardware changes during that time however. We are moving to a new big database server for Vista. Blackboard hosting will be making the switch for us December 28-30; Vista will be down for at least the 28th and 29th (beginning at 12:01 am on the 28th). We are not making any software upgrades or patches, so you shouldn't see any changes in your courses after the weekend. You might check anyway though, just to be sure.

The Banner student information system is also going through a major upgrade the weekend before Christmas, so Frank is coming in Christmas Eve to do some testing of our scripts.

We'll e-see you all in the next year!

Lisa

Vista Review of Recent Downtime

I wanted to give you a quick review of the past couple of weeks, since it may affect a few responses to students:

• December 1 and 2 - students were unable to submit attachments through Vista.
• December 3rd - BB re-booted the database and server (we have 9). This corrected the attachment problem, but meant BB was down from 11:30 am until almost 1:00 pm.
• December 7th: More attachment submission problems.
• December 8th: BB re-booted the system unexpectedly, down for approximately 25 minutes Saturday am.

That means there was at least one weekend when your students were genuinely unable to submit attachments and a couple of times when they may have been booted out of an in-process exam.

So what are we doing about this? We are upgrading to a snazzy new database server with more and faster processors. BB hosting has scheduled the transition to the new hardware for December 28 through 30. It should take 48 hours; when I have specific times I will let you know. You can assume that BB Vista will be down for most of the 28th through the 30th though. When we come up again we hope to be more stable.

Thanks for a good semester everyone! Technical issues aside we have once again had very few student complaints this semester. We truly do appreciate all of the time and effort you put into your classes!
Lisa

Grading Forms Bug - From ProfHelp

Faculty who are using Grading Forms:

If you are using Grade Forms in your course, please note that anytime you make a change to the Assignment/Discussion using the Edit Properties a new grade book column is created.

For example,

• Assignment 1 is gradeable using a Grade Form—an entry in the grade book is created named Assignment 1
• Choose the Edit Properties on Assignment 1
• Make a change to the properties of the Assignment for example change the due date of the assignment
• Click Save—a new column is made in the grade book and is named Assignment 1 1 (note a 1 is added to the end of the grade book column)

A possible issue with this scenario--The column containing the student grades may not be included within the Total points or a calculated column. Please review your grade book if you are using Grading Forms to assess errors related to this issue.

For assistance, please email profhelp@cccs.edu.

Amy Sorensen
CCCOnline-ProfHelp
amy.sorensen@cccs.edu
(720) 858-2731

Vista Bugs

Several of you have reported a problem with the discussion thumbtacks remaining after you have read all of the discussions. Unfortunately we haven't been able to replicate that one. Here is some advice from Profelp, Amy Sorensen:

I can not replicate the issue. I check the CRJ 125 course and after I read all of the discussions on my end the thumbtack and asterick in the Course Tools menu disappeared.

Maybe clear your Java Cache and be sure your browser is set to load a new page every time you visit a page. The instructions are below.

Let me know what you discover.

: Delete the Java Cache

1. Click on Start
2. Click on Control Panel
3. Double click the Java icon
4. Under the General tab and near the bottom of the window is a section called "Temporary Internet Files" --> click the "Delete Files" button
5. Close out of the settings window and the browser
6. Re-open the browser
7. Launch Internet Explorer
8. Go to the Tools Menu
9. Select Internet Options
10. From the General Tab, in the Temporary Internet Files section, Choose Settings
11. Be Sure the Every visit to the page is selected.
    

Lisa

Vista Update 082907

Hey Everyone- This note is going up on Blackboard Vista today:

Attention Students and Faculty: Temporary Service Outages

The Vista System will be taken down and brought back up daily at 2 PM, 8 PM and 4 AM for approximately ten minutes.

BlackBoard has identified the core issues with the recent latency in the performance. The reboot will be scheduled as a part of a short-term solution until we can schedule a software upgrade.

We apologize for this inconvenience. Please schedule your online activities around these daily system outages.

The splash page will be up during the brief downtimes and will include information about the duration of the down time.


What the cluster reboot does is reset all of the connections between the application servers (we have 8) and the database. Each server has 100 (rising to 125 later today) connections available. Apparently these are not being released properly once the current request has been processed. Re-booting the entire cluster (all 8 application servers and the database) resets these connections (releases them really) and allows performance to return to normal levels.

The root cause of the bottleneck is problems with the way Web Logic is communicating with the database server. The upgrade to application pack 2 includes an upgrade from Web Logic version 8 to version 9, which does include general fixes for this sort of problem. We will probably try to push that upgrade forward from the currently scheduled weekend (September 22-26), but have to complete some additional testing first. As soon as I have more solid information I will pass it on to you.

Thanks for all of the hard work! Students seem to be in their classes and happy with them. We are receiving very few support calls at this time and the support desk (Deb Michel) has been able to keep up with the calls and emails all week.

Lisa

Vista Update

We wanted to give you another update on Blackboard performance this afternoon. We are continuing to see some slow response times on and off throughout the day. This is a different problem from yesterday, and does appear to be application related as a result of very heavy use. We are working with Blackboard on possible solutions, one of which is to upgrade to AP2 sooner than our planned Sep 22 upgrade date. AP2 improves database performance, garbage collection, and also fixes a stubborn issue with Web-logic. We have authorized ASP to perform the upgrade on our QA server immediately so we will have an accurate estimate of the time needed for the upgrade. We have not discussed this with LTC yet, and will wait until we have a conference call with ASP at 6:30 this evening.

We realize (and have communicated to BB) that this is directly affecting courses system-wide, and will do whatever it takes to stabilize the system.

Lisa and Rhonda

Solution for Slowness

Hi Everyone-

Here's an update from David:

We had a phone call with BB today and covered several short and long term issues and goals. First, there is a team of DBA’s reviewing the behavior and performance of the Vista server. We should have their recommendations later today. Another action in regard to the current performance is to add another node with in the next 24hrs with a possibility of an additional one if needed.

Long term, we all know and agree to try to move to AP2 in the window we discussed, September 22. This is greatly needed to fix many of the issues that are affecting the performance of our bulging database.

Short term, we could get some improved performance if we could run garbage collection on Production. They have written a script and need to test this on the QA server first, then if that looks good, run it after hours on the Production server. Our concern is that we have our archives safe from previous terms. Toward this end, BB is setting up a third server and will clone Production there. Let me summarize the use and process of these three ASP servers:

1. QA server: this will be cloned from Production;
1. old terms (at least FA2006) will be deleted and garbage collection (GC) run here to test the script. This will probably occur on Monday.
2. After GC is run, the AP2 upgrade will be run and evaluated, some time later next week.
2. 3^rd ASP server: this will contain a copy of all of our data on Production.
1. If anything is lost at anytime over the next several weeks or months in making these adjustments, we will have a copy.
2. We also have a copy on the Test server, but it is not available as yet, due to some technical problems that we work on in our spare time.
3. In summary, we will have Fall 2006 and Spring 2007 in two places.
3. Production
1. If the GC script runs as expected on the QA server, they will do this after hours for several evenings on Production to remove deleted sections
2. LTC needs to determine what Terms we can live without on the Production server, so that we can enhance our performance

i. Can we delete Fall 2006?

ii. Can we delete Spring 2007?

One point that one of the ASP managers made is that the first week of online school for all of their customers is the busiest time of the term, but many students either drop or become less active after the first week. He thought that we had some advantage in not having all the colleges start on the same day but rather staggering them.

In summary, we are working with the ASP on resolving the current issue by adding an application node, possibly making some adjustments to the database operations, per the DBA’s recommendations. We are also taking an extra measure to previous term data, and initiating a series of tests and processes that will – we believe – provide better performance and a more stable software in a few weeks.

David

Just to update you on college start dates: we are slow this week because 6 colleges started classes yesterday. 8 others, including us, start this coming Monday. Vista will certainly be slow Monday and Tuesday of next week just because of that, although the additional node will help. Please be patient and don't plan on a lot of heavy class work until after mid-week next week.

Thanks-
Lisa

Update on Tonight's Maintenance

I have a little more news on tonight's downtime. This is what went out to students:

ACCESS TO CCCONLINE COURSES MAY BE DISRUPTED TONIGHT

Blackboard – the company that hosts CCCOnline classes – will make some critical repairs at its facility in Virginia tonight beginning at 10 p.m. The repairs probably can be done without disrupting access to CCCOnline courses. However, there is a chance that access to courses could be interrupted briefly.

To be safe, you should NOT start a test or online assignment that you cannot finish before 10 p.m.

The maintenance period could last several hours. CCCOnline will post a notice when the work is complete. You may continue to access their courses during the work, but be prepared in case there is an interruption.

If you try to login to a course and see a message about the repairs, you have been diverted to a different page. The next time you login, don’t forget to use this link: http://online.ccconline.org.

The key point is that we may or may not be down. Blackboard is trying to do this without taking us down, but that may not be possible.

The purpose of the maintenance is to replace equipment critical to the back-up system. Right now if we go down unexpectedly we don't have a current back-up. That is a fairly critical problem.

best,
Lisa

Emergency Shut Down Tonight at 10:00 PM

Hi Everyone,

We just received this notification from Blackboard hosting:

Dear Blackboard ASP Customer:

Blackboard Infrastructure will be performing emergency maintenance beginning tonight (July, 25, 2007) at 11:59PM EDT (9:50PM MDT) and continuing through 2:00AM EDT (12:00 AM MDT). This is in response to the unplanned extended outage which occurred on Friday July, 20. Blackboard ASP will be using this time to restore redundancy to your Blackboard environment. We believe the need to perform this work warrants the work being completed tonight rather than pushing this until our scheduled window of Friday morning. Although there is a risk associated with this type of maintenance we do not anticipate there being any downtime.

As a result, the redundancy testing that was scheduled to be performed Friday morning (July 27) which included known downtime, is being postponed until a later date.

If you have questions or concerns about this change, please contact your TSM. They will be happy to assist you.

Thank you in advance,

ASP Management

I will let you know as soon as I have more information. We are also notifying students, but if you have exams scheduled to end at midnight tonight you might want to consder adding an extra day. And don't forget to breath. :^)

Thanks-

Lisa

Removing_the_Calendr_Channel_From_Your_CCCOnline_Login_Page

Hi Everyone-

The calendar tool remains problematic for some Vista users. It's not causing problems for everyone, so we don't want to turn it off at a high level. However, some of you may want to turn it off for yourselves both within the course and on your MyCCCOnline page.

Directions for the latter are now on the Vista FAQ in the Faculty wiki at http://www.facultywiki.ccconline.org/~www66/index.php?title=Removing_the_Calendar_Channel_From_Your_CCCOnline_Login_Page
(That is a very long link, so you may have to cut and paste or go directly to the wiki at www.facultywiki.ccconline.org, click on WebCT Vista in the upper section of the page, then on Calendar tool. IT's the last link under Calendar.

The fixes to the calendar tool are currently targetted for an update to Vista in the future, so nothing soon. We are as yet unsure why there are problems for some and not for others, so are also still working on that.

best,
Lisa

Maintenance Window for Vista

Hi All-

From David:

I just got a call from our Vista TSM and there is a maintenance window scheduled for tonight at 1 AM MDT until 3 AM MDT. A message about this was supposed to be sent out earlier in the week for proper notification but that didn't happen. They are a little red-faced but they say they need to do this. They are doing some necessary maintenance for the entire data center, where we are hosted.

I hope you are all fast asleep in bed during those hours and were not planning on working in your classes then. Sorry for the late notice.

best,
Lisa