| Mail Server emergency maintenance |
| ( posted by Marc ) |
| Affects: | Mail Servers, and Webmail Servers |
|
| May 06, 2008: |
| 11:10am | | The mail migration has been completed.
All active accounts have had old messages, filters, and contacts migrated to the new server.
Any customers having further difficulties with their email should contact Support at 805-456-5800 or via the web problem report interface.
( posted by Marc ) |
| |
| May 02, 2008: |
| 12:00am | | The mail system will undergo a planned reboot at midnight, sharp. This should only result in a short 5 minute outage.
( posted by Victor ) |
| |
| Apr 29, 2008: |
| 05:10pm | | We have made the old email available through the webmail interface as we continue to port the old email directly into accounts. Users may go to https://webmail.impulse.net/ and select mail-archive.impulse.net from the "Server" drop down box and log in using their normal username and password. Note that any changes you make through this interface will be carried into the migration of your data onto the new server. Users can send email from that interface, but not receive, so our recommendation is to change as little as possible and forward any critical email to your email address so that it is received by your account on the new server.
( posted by Dave ) |
| 12:35pm | | Here is the current status:
OLD EMAIL THAT WAS STORED ON THE SERVER:
We were able to complete scripts on Saturday night to begin the migration of data that was stored on the server prior to the outage. We have four servers working on re-populating this data into the user accounts, however the same 12 million files that took two days to pull off the old disks is taking much longer since the email must be re-populated to active users accounts that are busy processing current mail. Although the pace of restoration is not completely predictable, our time frame for restoration is looking more like weeks rather than days.
We are going to change tactics and work on making the old mail available via web mail on a separate mail host system so that users can go in and access the old email. The old mail will be available on a separate system that you will be able to access via a web site with your current username and password. We will still continue the gradual process of migrating the email, however we are working to have the old email accessible by tomorrow morning.
WEBMAIL AND FILTERS/RULES FOR CURRENT EMAIL:
There is currently no webmail access for the affected accounts, and no access for users to re-build the filter rules that were in place on the old server. We are working with our programming team to determine how best to get these filters re-established, whether it is feasible to translate the rules from old system programatically or, if not, how to enable users to to create those rules on their own. We will communicate further progress here as we make it.
( posted by Dave ) |
| |
| Apr 28, 2008: |
| 09:35am | | Users having trouble sending only should adjust their email clients to use "SMTP Authentication" with NO SSL, if there is an option to "use same settings as my incoming mail server" that should be enabled. The authentication scheme is to send username (full email address) and password. "TLS" may be used, or else no encryption.
One last step is to change the outgoing server port (usually under advanced settings) to 587 .
( posted by Marc ) |
| 09:15am | | Users still having trouble connecting should check their email client settings:
customers with @impulse.net addresses most likely need to add @impulse.net to their username ;
@fix.net customers must use mail.fix.net for the incoming server name, and use their full email address as their username.
eg user@impulse.net or mail.fix.net with user@fix.net
These settings are usually under "tools" then "accounts" or "email accounts".
( posted by Marc ) |
| |
| Apr 26, 2008: |
| 08:20pm | | Thank you for your patience, we have begun restoring your saved emails back to your Inbox. If you have any questions please contact us.
( posted by Jbryan ) |
| |
| Apr 25, 2008: |
| 02:00pm | | Users who are still unable to login to mail successfully should call Impulse support at 805-456-5800 to have their passwords reset. Additionally, the restoration of missing mail is taking longer than initially expected to complete, however our systems engineers will be working around the clock throughout the weekend to ensure that no more time than is absolutely necessary is needed to complete this restoration.
( posted by Acalvert ) |
| 09:30am | | Unfortunately, intermittent time-outs and failed login attempts remain an issue at this time for some customers. Users who are able to login successfully may find folders missing but all previously saved mail in those folders has not been lost. These messages will soon be restored with the remaining messages that are also missing as the mail system is rebuilt. The majority of all affected accounts should still be able to send email. Restoration of services is still underway, more information will be posted as soon as possible.
( posted by Kinji ) |
| 06:00am | | Our tests throughout the night have uncovered an additional problem with intermittent timeouts. If you receive a timeout error when attempting to check your mail or send mail, please try again.
We are continuing to work diligently to resolve this issue as soon as possible.
( posted by Victor ) |
| 01:30am | | All users should now be able to access their accounts without getting their passwords rejected. Users who encounter any difficulties logging in to their accounts should contact our customer service group at 805-456-5800 (or toll-free at 877-692-2345). Inbound mail is now successfully being delivered, however, mail from approximately the past 36 hours is still being restored to user's mailboxes at this time. Further updates will be posted on this page as soon as they become available.
( posted by Kinji ) |
| |
| Apr 24, 2008: |
| 09:00pm | | The user authentication system is still undergoing testing by our systems engineers which has taken slightly longer than initially expected. Another update will be posted shortly.
( posted by Kinji ) |
| 05:30pm | | The new system has been built and the user accounts have been migrated. Queued mail is being delivered, however we are still working out the kinks in user authentication to allow users to begin checking mail. We expect to resolve this issues in the next two hours.
( posted by Dave ) |
| 12:55pm | | We will be re-building all users on the impacted partition on a new mail server. This will allow these users to receive mail that is queued and send and receive new mail. We are working to complete this process by the end of the business day. We will be working to restore the user mail that was stored on the server over the next 24 hours and will provide updates to that process as progress becomes apparent. There are 12 million files to be copied before we can start the restore, and that process is driving the lengthy timeline.
( posted by Dave ) |
| 10:25am | | Users on the 4th partition continue to be impacted. We have determined that the step we need to take to put the partition back in service quickly has the potential to create data loss. We are copying the data from the impacted drives before conducting this operation, however due to the the size of the data on the partition this will take hours. We are looking at alternative ways to get delivery of current mail working in the next 2 hours for impacted users while we work on restoring the stored data over a longer period of time.
( posted by Dave ) |
| 08:25am | | One disk partition remains in alert, affecting about 1/4 of users.
All users should be able to send email through their email clients, but not through webmail.
The users on the partition in alert will have their email queued at the Impulse mail server. When the partition is repaired, the queued mail will be accessible.
( posted by Marc ) |
| 08:15am | | Service has been restored to 3 of the 4 disk partitions and mail is being delivered to users on those partitions. We are working on the 4th partition which was the source of the failure.
Note that incoming mail for the period the server is/was down will be queued or bounced back to the user sending it.
( posted by Dave ) |
| 07:05am | | The maintenance is continuing and remains the highest immediate priority for the Systems Team.
( posted by Marc ) |
| 04:40am | | Emergency maintenance continues, thank you for your patience. We have performed necessary troubleshooting, identified failed components, and replaced hardware from spare parts. We are now in process of verifying all data.
( posted by Jbryan ) |
| 03:20am | | Emergency maintenance continues, thank you for your patience.
To clarify, Impulse.net mailboxes are unavailable. Checking email and sending email will not connect. All internet mail to you is being queued up until the maintenance is complete.
( posted by Jbryan ) |
| 01:00am | | The mail system has been taken down for another round of emergency maintenance. Engineers are working to restore service as soon as possible.
( posted by Victor ) |
| |
| Apr 23, 2008: |
| 04:45pm | | The server is back up and accessible.
( posted by Marc ) |
| 04:45pm | | The re-build of the disks using the disk system software was successful and the server is back up.
( posted by Dave ) |
| 04:00pm | | Two disks failed simultaneously on a system that is designed to continue operating with the loss of any single disk. We are attempting to re-build the disks using the systems' software. If that works mail service may be restored by 5PM. If not a lengthy hardware and software re-build will be required.
( posted by Dave ) |
| 02:55pm | | Our systems team is investigating a hardware alert. The mail server for @impulse.net , @fix.net, @pronet.net, @best1.net, and others will be unavailable.
( posted by Marc ) |
