BCT email disruption affecting some members
Update 7/7 @ 9:00 AM: We have received word that the planned repairs from the early morning maintenance were not successful. The engineering team worked throughout the night and restoration efforts are still ongoing.
Unfortunately, the affected email server remains offline and inaccessible for those users whose email boxes were hosted on this system. While we are thankful the issue is centralized to a subset of BCT members, we fully understand that this is a frustrating time for anyone impacted by the situation and extend our deepest apologies for this inconvenience.
We continue to explore every avenue to restore services as quickly as possible. More updates will be provided as additional information becomes available.
Update 7/6 @ 3:45 PM: Starting at 12:00 AM on July 7, email services will be unavailable for approximately 30 minutes while emergency repairs are performed.
While progress has been made in certain areas, some challenges remain regarding email services. The initial cause of the email disruption has been identified as server storage performance issues on the part of an upstream vendor partner. This resulted in two ongoing issues:
1. Mail processing became congested and some messages were delayed getting into user inboxes.
2. The mail server itself began operating at decreased capacity reducing the number of users that could access their email accounts at any given time - some people could get in right away while access for others was delayed.
Engineering teams had some means at their disposal for mitigating mail congestion and delivery times. They have continued to implement these measures and, at the time of this writing, all mail has been delivered to user inboxes.
Broader measures are needed to resolve the underlying server issues. To that end, engineering teams will be finalizing the move of some services to different physical infrastructure. This process will start at midnight on July 7 and continue for approximately 30 minutes. During this time, the mail server will be unavailable. No emails will be lost during this period. In addition, our vendor will have new server hardware delivered and installed on Wednesday, July 8.
With these combined changes, we anticipate an improvement in mailbox access over the next few days and a return to more normal operations later this week. We apologize again for the inconvenience this has caused our members. We thank you for your patience as we continue working diligently to resolve the situation.
Update 7/3 @ 1:30 PM. All backlogged email has been delivered as of 10:30 AM Friday. In the short term, our engineers continue to monitor the situation and will manually push any additional congested mail out to the intended recipients at regular intervals. For the long term, new equipment is in the process of being tested and installed which is expected to eliminate the issue moving forward. We recognize the importance of dependable email in daily interactions and are placing every effort toward its repair.
UPDATE 7/2 @ 4:15 pm. Work continues to free up the backlog of email and get it delivered to respective members. The majority of email accounts should be back up and functioning as of last evening. However, a portion are still slow to deliver and send. We will continue to post here as the situation changes.
UPDATE 7/1 @ 4pm: Our Engineering and Operations teams have been working night and day to resolve issues impacting our email service and have made significant progress. The backlog of email messages has been flowing throughout the day and users have been receiving their messages. We anticipate that normal email operations should be restored later this evening. We thank everyone for their patience and understanding as we tackle this unforeseen situation.
We are currently experiencing a disruption with our email server. This affects many member email addresses provided by BCT. Our engineering and operations groups are diligently working on restoring access to email services. We apologize for any service issues you may be experiencing regarding slow access. We will provide an update as soon as we can.
Go back to news feed.