Network Status News & Information

Server ultimate2 may be experiencing intermittent connectivity issues this evening during a server update.  This will resolve itself shortly.  We appreciate your patience during this process.

Server ultimate1 may be experiencing intermittent connectivity issues this evening during a server update.  This will resolve itself shortly.  We appreciate your patience during this process.

Server is rebooting and will be back online momentarily.

Accounts on servers host10, host11, host12, host13 will be migrated to a new more powerful server this evening at 7:00 PM MST on 02/15/2016. Some small amount down time may occur during DNS update and migration is taking place but should be minimal. Users whose accounts were migrated will recieve an email containing some new information regarding their accounts including a new IP address.  Thank you for your patience.

Server 15 is currently being rebooted after kernel updates.  Server should be back up momentarily.

Server 15 is currently being rebooted for maintenance.  Thank you for your patience.  This should take no longer than 5-10 minutes.

Ultimate1 is currently being rebooted for kernel updates. It should be back up momentarily. Thank you for your patience.

Ultimate1 is currently being rebooted for kernel updates. It should be back up momentarily. Thank you for your patience.

Server ultimate1 is currently being rebooted for maintenance kernel update.   Should only be offline for a few moments.  Thanks for your patience.

Server 12 is currently being rebooted for maintenance update.  Server Kernel is updating to Linux kernel.x86_64 0:2.6.32-504.16.2.el6.   Should only be offline for a few moments.  Thanks for your patience.

Server 12 is currently being rebooted for maintenance update.  Server Kernel is updating to Linux Kernel 2.6.32-504.3.3.el6.x86_64.   Should only be offline for a few moments.  Thanks for your patience.

COMPLETED

We are currently rebooting this server for security updates.  Reboot can take up to 10-15 minutes typically.  Thank you for your patience.

We are currently rebooting server 5... It will be back online momentarily.

We have been investigating issues on server 8 (host8) today and have found that the /home partition on the server has gone into a read-only state. This means that data can be read from the partition, however no new data can be written. We have checked the SMART data for the drives in this server's RAID and have found from the check that both are reporting old-age characteristics, and even some errors. This means that the disks should be mirrored and swapped out for new one.

Unfortunately this does require some down time to take the server offline to perform the mirror and disk swap. We are currently working to scheduling a low-traffic time of day (most likely late this evening) to perform this. We will update this post as soon as we have the exact time of the hardware swap.

***UPDATE*** 09/20/2014 15:15
Drive Swap is scheduled for this evening/morning September 21 at 3:00 AM EST. This could take between 1 and 3 hours depending on the amount of data that needs to be transferred to the new drive.

Accounts on servers host and host2 will be migrated to a new server this evening at 1:00 AM EST on 09/17/2014. Some down time may occur during DNS update and migration is taking place but should be minimal. Thank you for your patience.

We are aware of an issue regarding inability for some customers to access their Ultimate administration areas at this time. We are diligently working on resolving the issue as quickly as possible.

We are aware that there are some of our customers that are not able to reach some of our servers.
Rest assured that these issues are NOT directly related to any of our infrastructure.

You can test your website here and see that it is reachable by the world. --> http://www.viewdns.info/ismysitedown/

However, there are several ISP's (Comcast, Cogent, Level3, ATT, Time Warner, Verizon, and Charter, among others) throughout the United States that are currently experiencing issues. Even though our network is not experiencing issues, this is causing a large amount of latency to our network to the point that our network can not always be reached by all locations.

Also, please note, that even if you are not a direct customers of any of these ISP's, if any data normally passes through any of these networks between your location (or some of your customers) on the way to our servers, it may be shown as unreachable.

Our network team is still working to see if we can shift traffic over other providers, but it is not just one provider that is down, which makes this nearly impossible. Many ISPs have been affected by this as you can see on the maps tracking these large ISP & Network providers:

http://downdetector.com/status/comcast-xfinity/map/
http://downdetector.com/status/cogent/map/
http://downdetector.com/status/att/map/
http://downdetector.com/status/charter-communications/map/
http://downdetector.com/status/time-warner-cable/map/
http://downdetector.com/status/verizon-wireless/map/

We do not currently have an ETA on when this will be resolved, but we will be putting updates on our network status page as soon as we have more information from the ISP's being affected.

Important server MYSQL upgrade in progress......

The host5 server has recently shown signs of wear and aging in the past several months. We will be migrating the accounts on this server to a newer, faster, more powerful machine. To provide the best possible service to our clients, we must be proactive and complete this process now in order to avoid any potential hardware failures resulting in data loss. The migration will begin this evening at 1:00 am EDT (UTC - 5 hours) on Saturday morning July 12. After much analysis, we have carefully selected the lowest-traffic time period on the server, as to disrupt the website operations of fewest amount of clients. The migration may take up to 2 hours to complete. The migration will include some down time so sites may not be accessible during portions of the migration.

Clients with sites on host5 can begin to expect faster loading sites and fewer interruptions/slowdowns in service after migrating to the newer machine. Clients do not need to do anything on their end in regards to the migration. Name Servers and IPs will be swapped to the new machine and all services (http/email) will function normally after the swap. If you notice issues during the time between 1:00 and 3:00 EDT, please do not submit a support request as any issues are more than likely due to the migration. This will allow us to focus on the migration as opposed to answering support tickets regarding the migration.

However, although not anticipated, there could be residual issues related to possible differences in configurations, application versions, between the 2 machines after the migration is completed. If you notice anything abnormal after 3:00 EDT July 12, please submit a support ticket indicating specific details and instructions on how to replicate the issue so that we may investigate and solve any issues related to migrating servers.

We thank you for your patience during the migration. Should you have any questions, please feel free to ask.

Excessive mail queue on server 7 was discovered. Upon investigation, we discovered one user's account password was compromised and sending out large amounts of spam. Mail Server was stopped and we are working on clearing out large amounts of spam emails in the queue. This could take a while as the queue was quite large. Once the spam emails are removed from the queue we will continue processing the normal mail queue. Delivery and receipt of email messages on server 7 will be delayed until after the spam queue is cleared. Also connections to the mail server on 7 will not be successful. The user whose account was compromised has been temporarily suspended until the cause of the breach can be investigated to prevent further issues.

host5 has become unresponsive and is currently being rebooted. This process could take 5 - 10 minutes. Please do not open a support ticket. Your site will be back momentarily.

Due to software RAID issues on ultimate2 server, the server is being taken offline temporarily in order to swap out a piece of hardware. Thank you for your patience. We will keep you updated.

Due to Critical Security Updates announced by cPanel ( http://cpanel.net/cpanel-tsr-2014-0003-announcement/ ) we are required to update cPanel to the latest release. Because this update requires a higher version of MySQL than what is currently installed on the server, we need to do a mandatory MySQL upgrade. This will cause sites hosted on host12 to appear down for up to 10-15 minutes.

We are currently upgrading mySQL and PHP versions on Server 5. This will take about 10 minutes and you may experience momentary lapses in connectivity.

We apologize that the upgrade is taking much longer than anticipated. We are in the finishing stages and it should be back in working order momentarily.

We are currently seeing another DDoS attack on Server 4. We are currently investigating and will keep you updated as to when the attack subsides.

We are continuing to see small, brief DoS attacks on host4. We are attempting to narrow down the site that the attack was directed at. Additionally our networking team is actively monitoring the server and is working to intercept and redirect the malicious traffic before it has a chance to cause any outages. If your site is hosted on server4 you may still be experiencing small/brief interruptions of service. We will bring more updates as they become available. Thank you for your patience.

For customers on server4 (host4) we do regret that the server is currently under a large inbound distributed denial of service (DDoS) attack of size 10Gb/s. The attack appears to have been targeting the server's main IP address. Whether a website on server 4 or server 4 itself was the intended target, we are not sure. Unfortunately at this point little can be done to stop the attack and we will keep you updated as to when the attack subsides. To find out more about DoS attacks, read here: http://www.digitalattackmap.com/understanding-ddos/

1. *cPanel Password Resets* - Part of this security hardening process includes cpanel password resets to meet current strong password requirements.
   
   We are working on getting new cPanel password emails out to all of our customers as quickly as possible. We hope to have all of those out by the end of the day today or tomorrow. In the meantime, if you require access to cPanel or FTP and are having trouble connecting, please open a support ticket and we'll generate your new password reset email as quickly as possible.


2. If you **think** your site is down, it is probably not actually "down" for everyone. If it appears down to you, it is very likely that your own local IP is blocked in the firewall due to the password changes mentioned above. Check this link to see if your site is really "down" or if it's just you:
   
   Is My Site Down?
   
   You can actually unblock yourself by following those steps in that article.

Thank you for your patience as we transition our servers into a more secure server configuration.

Just to keep all of our valued customers in the loop we wanted to make you aware of some server activities today.

All servers are currently undergoing a thorough Security Audit and Server Hardening process in order to provide the greatest security possible to avoid any exploits or outside intrusions. You may experience brief sporadic connectivity issues throughout the day as servers are being brought up to date with the latest software and kernel updates and security upgrades. But please know that we are aware of any issues and are doing our best to keep any necessary interruptions very brief in nature. We thank you for your patience as we work to improve overall server security for all of our customers.

During a routine Apache update, the server encountered some unexpected errors. Our technicians are currently working to bring the server back online and we anticipate this should not take very long. Thank you for your patience.

Server 10 is currently being rebooted and will be back online shortly. Thank you for your patience.

Apache has stopped responding on server 10 (host10.shoppepro.com).  Apache is being recompiled and will be restarted shortly.  Unfortunately, this process can take 30 minutes or more.  Thank you for your patience while we work to restore service.  

Server 10 is currently under a DDoS attack -- http://en.wikipedia.org/wiki/DDoS_attack.  We are currently working with the server admins to bring down the server load.  Thank you for your patience while we deal with this issue.

Server 6 will be rebooted shortly and a short amount of downtime will be experienced during the reboot.  Saturday June 26, 2010 12:05 pm MST

We will be performing a mysql database optimization at 6:00 am EST (U.S.) on January 22.  The optimization will take about 30 minutes and might cause some database outages but it will be back up and running by 6:45 AM .  If you have any concerns, please let us know.