Open Source in a Sales Management System

NOTE: This short study is a sketch that hardly does justice to the scale of the project — but for commercial reasons most of the details have had to be omitted.

We've long been believers that Open Source products outperform most of their closed-source competitors, even when applied to commercial line-of-business applications. In line-of-business applications, reliability and robustness are the watchword and the systems have to work. Our experience of key components such as Apache and Perl lead us to believe that they are better and safer bets than many of the allegedly commercial alternatives.

We were approached by Bryant Homes in mid 1997 to discuss the possibilities of building a distributed sales tracking and processing system. Bryants are one of the leading UK house builders (here is their website) and like most house builders they have a particular problem: their shop front is mobile and most of their customers come to where they are building the houses. A building site will be active for a certain period before all the plots are sold and the "shop" then is itself sold (the show house): at that point the sales operation is closed down for that site.

A central database is used to track the sales operation, located at their head office in Solihull, West Midlands. It's not realistic to put in expensive fixed lines to temporary offices like the show homes, but obviously it's very important to be able to automate the workflow of the sales process. In fact it's particularly important, because many of the staff who operate the process work part time and need support in managing the continuity of the relationship with their customers.

In mid-97 it was clear that something could probably be done using web technology and making use of a browser as a front-end. Using standard components like those, dial-up access back to head office for the data transactions and some intelligent use of local storage on the sales PCs (images of houses, plans, non transaction-related data) a highly cost effective solution could be built. A crucial issue was to keep the connectivity costs down: initial figures showed that a year's phone bill would dwarf the development budget if all of the sites maintained 8-hour connections. There are a lot of sites out there!

Fairly rapidly an architecture evolved. This general architecture is very well understood nowadays, but remember we were working way back and were treading a much less smoothly-worn path.

The Head Office Database is a large Informix database running on HP 9000 systems. A separate system runs Apache on HP/UX, where the queries to the database are fielded by a FastCGI process which handles those of the CGI scripts that are written in Informix embedded SQL. Fast CGI was necessary because of the delays introduced by connecting to the Informix database; for some reason this operation consumes a lot of time. Fast CGI allows a permanent connection to be maintained and shaves seconds off the creation of each page.

The Informix SQL queries were deliberately kept simple and were based on the analysis of the transaction needs of the system as a whole. They provide a data channel only — the work of making individual queries for specific pages and formatting the results is done by extensive Perl scripts running in Apache's mod_perl service.

A lot of development work went into creating the "build tools" as they are known; these test that the production Perl scripts are syntactically correct and validate the parameters that pass between the Perl and the SQL scripts. The build tools consumed a significant amount of early time in the project but proved their worth when it came to building the deliverable software later.

The on-site systems use Netscape running on Windows. A local webserver and CGI scripts are installed on these systems and automatic uploads of data to the PCs are controlled from the Head Office systems using opportunistic "push"; when the on-site system connects via ISDN, stale data is refreshed from Head Office. Many on-site queries can be resolved by referring to local data (site plans, images of houses and numerous others), so the connection to Head Office is not kept permanently open. From time to time key transactions do occur on the on-site system and these are pushed over to the Head Office systems, causing a connection to be made.

Connections from site to Head Office are via ISDN line, received by a standard Access Port / RADIUS server combination as found in any number of ISPs.

The on-site systems make very heavy use of Javascript to provide as much front-end validation as possible. The web browser is heavily customised (hence the choice of Netscape) to remove the chances that the end-users will press the "wrong" buttons or go exploring; in essence the web browser has been converted to "kiosk" mode. This was no easy feat and involved signed Javascript and a lot of fiddling about. We did finally manage to lock it down pretty tightly and this has resulted in very few surprises in the field. It's worth emphasising that many of the sales negotiators are not technically literate — so they need a system that is very easy to use, and that they also get bored if there is not too much to do on a slow day and might be tempted to fiddle with the system. Fiddling about is OK so long as nothing breaks - support staff hundreds of miles away really don't need the problems caused by trying to debug a remote setup that has had its proxy settings altered (for example).

An important part of the design was to store no state at the remote end. Building sites are notoriously prone to theft and the PC systems installed there need to be as hot-swappable as possible. The design successfully achieved the goal apart from the issue of the remote IP address, which is used for validation of incoming calls at the Head Office end.

Three separate servers are used in the development system, listening on different TCP ports. The standard production server listens on port 80; others on port 81 and 82 are used for pre-delivery testing and development in parallel. The software configuration system (based on RCS) is aware of this and tools are used to automate building the various deliverables as well as performing regression testing.

In general the Open Source tools have been absolutely problem-free. The notable exception was the use of Hypermail as the tool to convert inter-site email archives into web-browseable archives. Considering the pain and grief that Hypermail caused us, it would probably have been quicker to have reimplemented it from scratch rather than try to make it work on the scale and volume of messages that it was asked to handle. Several tense project management "moments" were the direct result.

That apart, the project has been a success. It went into production in 1998 and has proved popular with its users from the beginning. Development has continued since then, with increasing demands for further functionality. It's hard in a short study like this to give a feel for the scale of the product — this is not a little "demonstrator" project, it is a full-scale mission-critical application that has had a direct impact on the business processes of its user.

As a footnote — even more problematic than Hypermail was the decision to implement some bells and whistles in Java (as applets) in subsequent phase. The experiences that we had of that have left us with an even lower opinion of Java as a serious delivery tool than Javascript - and our opinion of Javascript is well-nigh unprintable. Perl and Apache though: what a combination!