October
21st
Dear DJ Webmin Clients,
For the last 48 hours, ProDJ.com has been working on recovering from the
largest loss of data in our history. We have spared no expense
finanacially or in the form of time of our technicians who have worked
around the clock since Monday night at just before midnight, when the
Microsoft security updates were installed on that database server and caused
its crash Start.ProDJ.com and Support.ProDJ.Com are now operating
under a database from mid-August that was pulled out of the datacenter
for testing purposes. DJ Webmin was not so lucky.
DJ Webmin's data backup procedures were in place and were backed up every
morning. The problem is that due to the crash all complete file copies
of DJ Webmin data are gone, and the incomplete ones do not have enough to
even make it worthwhile to reconstruct... it is less than 2% of the data
since April's database work. However, we have restored data from
that date (April 2nd) and previous . (Any data entered
after that date is gone.) While I still have complete faith in my
technicians responsible for our backup procedures, I share the
responsibility with them in that ProDJ.com is my company, and
that I am a partner with the developer, Barry Cedergren in this valuable
service. DJ Webmin will survive. It is a fantastic
product/service that many many people (myself included) trust with
their business and
is one of the leaders in the industry.
We are asking for your trust in
DJ Webmin and its development team in making DJ Webmin back into the service
that many businesses trust. What we would like to offer
you is the following:
1) Anyone that is currently with us on the service will have
their subscription extended by a full year. Additionally, new
subscribers that sign up for DJ Webmin Lite or Pro by November 1st will get
their account at no cost to them for one year; just fill out the trial
account and it will be set up as a full account.
2) Pending whatever data that you have, we will help you
re-enter it into DJ Webmin. If you had taken advantage of the export
procedures that are recommended with use of DJ Webmin, we ask that you
send those excel data files to
recovery@djwebmin.com along with your user account number and password.
We have software in place to import the data for you. If you do not
have any backups of your DJ Webmin data, we will work with you as much
as possible to get the data re-entered. Please email
recovery@djwebmin.com with your account number, number of events you
should have in the system and any other information that will help us serve
you in this process.
For those of you who subscribe to DJ Intelligence in addition to DJ Webmin,
there is positive news! As you may know, DJ Intelligence is a
completely separate company and service from DJ Webmin. As such, they have
independent servers in their own data facility in Chicago. Your
client information, including request lists, planning forms, event
timelines, guest request lists, satisfaction surveys, client logins, and
more are completely intact and 100% accessible directly through DJ
Intelligence. This data is stored directly on their servers and was
uninterrupted throughout this entire incident. Simply log into your DJ
Intelligence account at
http://www.djintelligence.com/account and select the Client Sessions
area of your control panel to access this information. In addition, if you
save your email, you will be able to go back and review all of your
inquiries, quotes, payments, and bookings that came in through their
system via email. It is important to stress that DJ Intelligence, being a
separate company, was not affected whatsoever by this outage and you can
feel confident in continuing to use their tools on your website independent
of DJ Webmin. More information about their network, for your peace of mind,
can be read at
http://www.djintelligence.com/pages/network.asp.
What has been done to prevent this from happening in the future?
We have started 3 levels of backup procedures that will leave us completely
covered by a "security blanket" should any issues arise again.
We felt that the drive array and full backup system were comprehensive
and sufficient; however, we have gone two levels past that in a new
backup program that will be commencing tomorrow, October 21.
Additionally, there will be more ways for you to keep your data offline
within the near future. Watch your email for information on this
upgrade.
ProDJ.com is well known within the disc jockey industry as the leader in the
field of Internet services for DJs, and we will continue to innovate and
serve the disc jockeys that depend on our services. Look for bigger
and better things from "The Internet's Source For DJs" as we learn and
improve from this issue.
This is a very unfortunate situation and has happened to the best of
companies. I can think of examples when many great service providers were
down a day or two in the past and/or lost recent data due to unforeseen
circumstances. Most of us have at least one insurance policy
in place; you hope you never need to use it. All one can do is be as
prepared and comprehensive as possible. Should a day arrive when
something happens beyond what your most thorough planning could anticipate,
you do the best you can and work as hard as possible to get back on track.
Then you learn from your mistakes and put new measures in
place to prevent it from EVER happening again. And tonight, I can
faithfully make that promise to you.
Thank you,
Ryan Burger
President
ProDJ.com Internet Services
October 22nd
DJ Webmin is back up for you to login! This is the current status of
the recovery process:
1) As mentioned in the email of late October
20th, all data has been lost since April when we made the conversion to a
new database structure. All data including logins, packages, events, etc in
that system are active now. Any employees set up since then will have
to have their logins re-setup.
2) Anyone who came on with DJ Webmin after
that date has had their base account (without any bookings or Administrative
Options) re-setup. You will need to use your password you had from
April in DJ Webmin. If you have trouble
logging in please put in a support ticket immediately at
http://support.prodj.com, and one of
our 4 techs on duty for this morning will assist you!
3) If you have data that you would like
ProDJ.com to work with you on we are offering to assist with data entry of
DJ Webmin
contracts or Event info forms that you have printed out, Excel spreadsheet
backup files or saved data in some other form. Electronic forms of data you
have can be emailed to
recovery@djwebmin.com, be sure to include your DJ Webmin account number
and password in the email to identify you. Or if you have hard copy data you
can fax such info to 515-986-3344 with your username and base password on
the cover sheet or overnight express them to 807 N Park St, Grimes, IA 50111
or send via normal postal mail to PO Box 42365, Urbandale, IA 50323.
4) The 3-level backup procedure now in place
will be announced in detail on Monday, October 25th, 2004. The procedures
for back-up will be extreme and will contribute to our goal to never let the
possibility of total data loss ever happen again.
5) DJ Webmin is planning to develop some
exciting new features that will make using DJ Webmin better than ever
before. These features will include better data backup processes on the user
end and even more useful tools as well as other features we have gathered
from user suggestions.
Overall we have had very supportive response
from DJ Webmin users in personal conversation and through posts on Start.ProDJ.com and DiscjockeyAmerica.Com and
appreciate the votes of confidence. Even though your expectations were let down in a
large way with server side data recovery, most users we have been in contact
with are ready to be productive with getting
everything back off the ground. I have already received dozens of
emails to that effect and in my phone calls to over 40 clients so far in the
last 24 hours, the support has been positive. We have a large group of users
who are some of the best in the industry, most have had an implemented local
backup system and will find this recovery process fairly simple. We want to
do the best we can for everybody and are very thankful for everybody who has
let us know they understand our commitment to DJ Webmin and their business
success.
Thank you,
Ryan Burger - President ProDJ.Com
Note, we will be making updates to the DJ
Webmin control panel location of this information for the foreseeable future
and will stop emailing after Monday's update as it currently stands.
October 25th
Of the DJ companies that we have spoken to or
traded emails with, the vast majority (like 90%) of them are recovering well
from this situation. Either by re-entering their data or considering other
services/programs to take care of their needs. And the vast majority of
those are sticking with us. It is amazing how loyal companies are being to
our service, and we thank them for doing so. Within the next 6 months, our
development schedule has new features and benefits that will debut that will
earn many of those other companies back to DJ Webmin...stay tuned!
We are doing our best to address all of the situations and special
circumstances that different businesses have and hope to have the remaining
10% of the businesses effected by the data loss resolved as best as possible
within the week. As offered previously we have arranged for data entry
specialists to take and enter contracts for you.
Please see last Friday's update for more
information. We are also offering totally free accounts to any other
companies interested in our service for the next year as long as they sign
up by November 15th (thanks to the over two dozen who have within 48 hours
of the offer being made). Current clients are either having their past
invoices (that haven't been paid since invoicing this year, or their
upcoming renewals extended by 1 year.
A new offer discussed at our DJ service meeting last night is that ProDJ
will help you throw a "data entry" party of you are having one like we did
last night. Tell us when and where, and pizza and refreshments (if within
the delivery area of one of the major chains) will be delivered. Our
session last night with practically all of our staff in attendance
went very well and served as a great opportunity for camaraderie. We want to
work with you to do whatever it takes! Data Recovery Suggestion: * Check the
cache of your internet browser. To find out what to look for and what to do
to recover this information, check out this thread on Start.ProDJ.Com:
http://start.prodj.com/viewtopic.php?t=18285
As promised our backup plan:
1) What we had before was a backup being done at 4am every morning and
stored on the raid server. Occasional copies were being made and stored out
of the facility (those were the ones that also failed). All of this was done
in compliance with our Terms of Service. The problem was that the files
failed us.
2) After an intensive discussion with our staff and outside data reliability
services, this is what we will be doing for now on:
DJWebmin, Start.ProDJ.Com, Support.ProDj.com, and all other Dynamic
Databases stored on our master server:
These databases will be backed up beginning at 4 am daily. This will be done
in a 3 step process. At 4am, a copy of each database will be made on our
database server. This copy is being made so that users working on our pages
will not have their speeds affected for when the later backup steps are
being performed. At 4:30, that file will then be copied to our new dedicated
backup server located within our offices. At 7am, when our support shift
starts, the files from the backup server will be burned to DVD-R. When our
support staff shift change occurs over the lunch hour, the DVD-R will be
archived to a location outside of the ProDJ.com data facility. The DVD-R
discs will be returned to the ProDJ.com offices after 3 weeks for
destruction for privacy purposes.
Other websites and hosting clients:
Currently, ProDJ.com hosts over 500 sites on our web servers. With the
amount of data space and time needed to back-up that many sites, we will be
doing backups of web hosted sites on a weekly basis. This will keep users of
your sites from being affected by any potential lag time of your site while
it is being backed up. The steps for performing the back-ups will be done in
the same manor and procedures as the database
backups.
Once again, DJ Webmin is recovering thanks to the help of many people,
including but not limited to the users of the software who are
overwhelmingly sticking with us through this, the moderators and regulars at
Start.ProDJ.Com, the development and support team at ProDJ.Com including
Barry Cedergren, Dwight Porter and Terry Legge, our
friends at DJ Intelligence, and our competitors at WEDJ's GigBuilder system
including Scott Rosenberg and Sal Pitera.
Thank you all for sticking with us....please address issues to our support
boards at http://support.prodj.com
as I am still overwhelmed by emails and phone calls. I have returned over 70
calls in the last 3 days and have about another 20 or so to go, as of the
last time I checked my voicemail. If you haven't received your call back
yet, please be patient as I should be caught up by the end of the day today
(Monday) with all calls that came in over the weekend.
Ryan Burger
ProDJ.Com Internet Services. |