I would like to automate our company's process of releasing new web functionality into production.
Here's a brief description of our architecture:
- RH Linux
- MySQL database on the back end.
- All HTML and Perl source code is stored in a CVS repository. Images are not stored under CVS.
- Each developer has their own local source tree with an Apache server listening to a high-numbered port.
- We also have a staging server that we use for testing
- We have a separate production server visible to the outside world.
- HTML pages are static files.
- We write mod_perl handlers and cgi scripts that run under Apache::Registry.
At its simplest, you could imagine a release process scheduled to run every night at 1:00 am that looked something like this:
cvs update
/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl start
However, this approach is likely to prove unsatisfactory for several reasons:
- Since images are not stored under CVS, new images might not get loaded when needed.
- Some development involves creating new database tables, altering existing tables, or populating existing configuration tables with new data.
- New modules should be tested before stopping the web server. Should a module fail some of its tests during installation, the update should be rolled back so that the server can start successfully.
- Some files get committed to the repository before they are stable and/or approved. Only stable, approved files should get released.
- A related (but distinct) issue is that stable files should not necessarily be released as soon as they are stable. A simple example of such a case is a (stable) HTML file which includes a link for another HTML file which is not yet ready to be launched.
Given this description, it is possible to rough out a simple spec for The Release System (TRS):
- TRS needs to interact with CVS. This will allow releases to be customized by directory hierarchy, CVS module, and / or tag. This will also ensure that master copies will be released, rather than local copies.
- TRS needs to be able to execute simple scripts. This will allow TRS to run tests, execute arbitrary SQL, log results, etc.
- TRS needs to be able to perform a rollback should unit tests fail during release. It should only stop and start the web server if all tests pass.
- TRS needs to interact with some system that knows (1) whether a particular file is stable; and (2) whether a particular file is approved for release.
- TRS should have logging facilities that keep a record of when all changes were made, error messages and warnings that were encountered, etc.
- Although a fully automated and regularly scheduled system is nice, it is not necessary. However, a release should be possible with very little interaction on the part of the system administrator.
- TRS should be able to run the regression testing suite after release.
A few questions:
- Who has cracked this nut before?
- Can anyone recommend pointers to good articles that tackle part or all of this?
- make is certainly a candidate for some of the logic -- would anyone recommend a different installation utility?
- Parts of this sound like a Content Management System -- does anyone care to comment on the applicability of Bricolage or a similar open source CMS product?
- CVS does not seem to have as robust a system for tracking stability as I would like -- i.e. you can set tags on groups of files, as in "STABLE-06012002", but you can't flip a bit per file. (Unless you set a tag per file -- which isn't necessarily good since a file might go through various iterations of stability / instability.) Can anyone venture an opinion as to whether the per-file "stable" flag is a good idea or a cluster-f*ck?
Thanks for any and all help.
