BCT email disruption affecting some members

Posted by Conner Williams | Jun 30, 2020
#outage #announcements

UPDATE 7/8 @ 4:30 PM


We are thrilled to relay reports from our vendor that BCT email services are back to functioning properly. New equipment has been installed and all messages have been delivered. Additionally, email access through POP and IMAP is up and running, so those that download messages to a desktop computer, smartphone, or other device through an email program can resume normal usage. 

The temporary webmail link provided yesterday as a workaround will only be active for another 24 hours for safety measures.  The engineering team will continue to monitor operations over the next several days to ensure platform stability.

We realize the last few days were quite frustrating for many. Our entire team would like to sincerely thank our members for their patience and understanding during this difficult situation. Significant steps have already been taken to improve the long-term stability of BCT email services, and we have begun the exploration into alternative infrastructure solutions. 


--


UPDATE 7/7 @ 4:30 PM

If you are having issues accessing BCT email, use the alternative login in the yellow box provided on the webmail login page here: https://bit.ly/BCTwebmailYou will need your full email address and password to access the portal.  

We have a little bit of good news! We’ve been told by our email vendor that the server is back to semi-operational. However, it is unfortunately not fixed, just temporarily patched. Affected members will continue to see intermittent delays and congestion in the current environmentThat is not the experience we want for our members. So, there is a new plan to restore full functionality 


POP/IMAP Email will be temporarily suspendedTo fix the problem, we will need to temporarily turn off local client access to all affected email accounts for up to 24-hours. That means, if you download your BCT email to your desktop, phone or other device through an email program, your access to receiving and sending messages through this method will be temporarily suspended


Web Access to Email is availableMembers may still read and respond to BCT email through the Webmail portal during this time. This means you’ll need to use an Internet browser like Chrome, FireFox, Safari, Edge, or Internet Explorer to access your email. Our vendor has provided an alternative link in the yellow box at https://bit.ly/BCTwebmail to see and respond to email in the interim. Please have your email address and password handy before you visit the link.  

Email read on Webmail will still be delivered to the desktop client(s) when all is fixed. If you are new to accessing your BCT messages via webmail, rest assured that your messages will still download to your usual desktop client when everything is back to working as it should. 

Your messages are safe. We have been informed that to date, no emails have been lost but are simply in a holding pattern waiting for the replacement system to be fully operational before delivery can occur. Assuming this effort is successful, affected users may receive duplicate emails as the team will be send everything in the queue. 


--

Update 7/7 @ 9:00 AM: We have received word that the planned repairs from the early morning maintenance were not successful. The engineering team worked throughout the night and restoration efforts are still ongoing.

Unfortunately, the affected email server remains offline and inaccessible for those users whose email boxes were hosted on this system. While we are thankful the issue is centralized to a subset of BCT members, we fully understand that this is a frustrating time for anyone impacted by the situation and extend our deepest apologies for this inconvenience.

We continue to explore every avenue to restore services as quickly as possible. More updates will be provided as additional information becomes available.

--

Update 7/6 @ 3:45 PM: Starting at 12:00 AM on July 7, email services will be unavailable for approximately 30 minutes while emergency repairs are performed.

While progress has been made in certain areas, some challenges remain regarding email services. The initial cause of the email disruption has been identified as server storage performance issues on the part of an upstream vendor partner. This resulted in two ongoing issues:

    1. Mail processing became congested and some messages were delayed getting into user inboxes.

    2. The mail server itself began operating at decreased capacity reducing the number of users that could access their email accounts at any given time - some people could get in right away while access for others was delayed.

Engineering teams had some means at their disposal for mitigating mail congestion and delivery times. They have continued to implement these measures and, at the time of this writing, all mail has been delivered to user inboxes.

Broader measures are needed to resolve the underlying server issues. To that end, engineering teams will be finalizing the move of some services to different physical infrastructure. This process will start at midnight on July 7 and continue for approximately 30 minutes. During this time, the mail server will be unavailable. No emails will be lost during this period. In addition, our vendor will have new server hardware delivered and installed on Wednesday, July 8. 

With these combined changes, we anticipate an improvement in mailbox access over the next few days and a return to more normal operations later this week. We apologize again for the inconvenience this has caused our members. We thank you for your patience as we continue working diligently to resolve the situation.

--

Update 7/3 @ 1:30 PM. All backlogged email has been delivered as of 10:30 AM Friday. In the short term, our engineers continue to monitor the situation and will manually push any additional congested mail out to the intended recipients at regular intervals.  For the long term, new equipment is in the process of being tested and installed which is expected to eliminate the issue moving forward. We recognize the importance of dependable email in daily interactions and are placing every effort toward its repair.

--

UPDATE 7/2 @ 4:15 pm. Work continues to free up the backlog of email and get it delivered to respective members. The majority of email accounts should be back up and functioning as of last evening. However, a portion are still slow to deliver and send. We will continue to post here as the situation changes.

--

UPDATE 7/1 @ 4pm: Our Engineering and Operations teams have been working night and day to resolve issues impacting our email service and have made significant progress. The backlog of email messages has been flowing throughout the day and users have been receiving their messages. We anticipate that normal email operations should be restored later this evening. We thank everyone for their patience and understanding as we tackle this unforeseen situation.

--

We are currently experiencing a disruption with our email server. This affects many member email addresses provided by BCT. Our engineering and operations groups are diligently working on restoring access to email services.  We apologize for any service issues you may be experiencing regarding slow access.  We will provide an update as soon as we can.  



Go back to news feed.