The Electronic Mail Service at UWO Peter Marshall, Info Tech Services, UWO March 26, 1996, v1.17 This document is also available on-line in hypertext format via the World Wide Web as . 1 Background and Summary At an ITS management meeting on 1995-11-10 the future of Emc2/TAO was discussed. At that meeting we decided that ...it is unlikely that we will have and support only one e-mail package on campus. The university community would not accept the idea of being "forced" to use only one package. The ideal will be to offer a suite of e-mail packages that are inter-operable and fairly easily supported. To aid in moving toward that ideal, a report was commissioned which reviews and rationalizes the packages currently supported and their uses/inter-operabilities. This is that report and more. E-mail has evolved into an essential core communications service at UWO. With the imminent shutdown of the Emc2 and VAX/VMS systems we conclude that an experienced, cross-group team needs to be immediately established to investigate future e-mail system possibilities and directions for the campus. 2 A Brief Chronology of E-mail at UWO E-mail at Western has had a long and distinguished career.(1) The following summary provides some perspective on how we reached our current state. That state will be described in more detail in later sections. ------------------------- 1. Thanks to the pioneering and continuing work of Reg Quinton as well as significant contributions from Colleen Bretzlaff, Eva Placko, Debbie Jones, Elizabeth Watt and a host of others at ITS. 1 late 70s Single computer, e-mail system on the DECsystem-10 timesharing system early 80s the MLnet project linked campus computers together using serial links --- fed e-mail into the X.400-based CDN network as well as having UUCP-based links (both via slow serial dial-up connections) mid 80s BITNET connectivity added, IBM PROFs system installed on central VM/CMS system for administrative e-mail. MLnet was used to gateway mail between campus machines. 1988 Internet connectivity established May 1992-- March 1993 CEDAR (Campus E-mail for the Desktop Architecture Review) group --- a joint CCS/DAS group that set out to ...to describe a campus-wide system that we could use to unify and simplify the provision of e-mail services at UWO. (From Focus volume 8 number 1, Electronic Mail for the 90s, March 1993.(2)) The group quickly discovered that this wasn't possible, but that a suite of ``state of the art'' programs could and would be supported to usher in a new era of functionality, Internet-compatibility and ease of use to replace the line-oriented mail programs with their roots in the 1970s. The recommended programs: o Pine; o two POP3 mail clients: NUPOP: E-mail for the PC, and Eudora: E-mail for the Mac; and o Pegasus Mail & the Charon Gateway(3) for users of Novell LANs. April--October 1993 Administrative Systems tested and then introduced Emc2/TAO via an article in the Information Exchange newsletter: Emc2/TAO - A New Email System, ------------------------- 2. URL: gopher://gopher.uwo.ca:70/11/.publications/Newsletters/Focus/- Volume%208%20%281993%29/V8.1%20%28Winter%201993%29 3. Ibid 2 September 1993.(4) By in large, most of the conclusions of the 1993, CEDAR group still stand. At that time the group studying the issue was ...confident that there is no commercial software that fits smoothly into the Internet environment that is as good and cost effective as the programs described in the articles [in the Focus issue cited above]. The basic conclusion that open, standards-based, post-office protocol version 3 (POP3)-based mail systems should be the ones that we move toward still represents our current direction. With the release in 1994 of the MS-Windows version of Eudora, we have consolidated on Eudora for both the MAC and the PC platform while the use of the DOS-based NUPOP has fallen away as the adherents of the non-windowing world disappear. The LAN-based solution of Pegasus/Charon has evolved into Pegasus/Mercury with Mercury providing POP3-based services as well as a gateway from the LAN-based Pegasus system. 2.1 A Note on Emc2/TAO Emc2/TAO was chosen as the replacement for the existing VM/CMS-based PROFs system for a number of reasons: o The VM/CMS machine was leaving and most administrative users were using the MVS system, so it made sense to provide a mail system on that machine. o There was an assumption that MVS was a long-term platform for administrative computing (and it probably will almost make it to 1999). o An e-mail system should include scheduling, conferencing and other office automation functions (the TAO or Total Automated Office part) o The vendor support implied by a commercial product. (Unfortunately, this didn't materialize). Emc2/TAO has never lived up to its initial promise, the promises by the vendor to provide a more standards-based system and continues to be plagued by operational and compatibility problems. Nevertheless it is widely used by many people throughout the University, especially the non-academic areas. Many of the non-email features of Emc2/TAO, like scheduling, have been widely praised. -------------------------4. URL: gopher://gopher.uwo.ca:70/00/.publications/Newsletters/- Info%20Exchange/Info%20Exchange%3a%201993/09%20September/114 3 3 Mail Systems on Campus There are basically four types(5) of mail systems represented by the approximately 136 mail servers(6) currently registered on campus: o LAN-based (21), o POP3-based (51), o host-based (80) and o proprietary client/server based (2). These mail systems provide e-mail service to about 3,300 of the 6,750 faculty and staff (or about one half)(7) and an estimated 8,000 students (under a quarter of the total). It should be noted that since e-mail is still by no means universal we cannot yet eliminate other forms of communication in its favour. In this section we'll briefly describe and distinguish each of these, itemize the specific implementations currently in use on campus and provide a cursory glance at the use that is being made of each system. 3.1 LAN-based E-mail LAN-based systems use the file system of a LAN server (like a Novell server) to transfer mail between users of that server. They typically employ an SMTP gateway to import and export messages to other systems. By far the most used LAN-based e-mail system on campus is the freeware package Pegasus using Mercury as the SMTP gateway process. (Older versions of Pegasus used a stand-alone PC for its SMTP gateway known as Charon, but these are pretty much gone from the campus now.) There are a number of commercial mail packages that use a similar approach, but on campus we are aware only of the Microsoft Mail System in use at PTCE. According to their support technician, Charles Deschenes,(8) they'd be happy to upgrade that to something that would use a POP3 approach as the Internet gateway for their system is ``flaky''. The teaching hospitals and the Addiction Research Foundation (ARF) use commercial LAN-based e-mail systems (primarily ------------------------- 5. with some fuzziness and blending in the classification 6. The number of mail servers on campus continues to grow. In 1988 we had 19, in 1992 we had 80 and now we have almost 140. We should remember that while many of these machines are single user workstations some are large servers like julian/mustang with over 9000 registered users. 7. These numbers are extracted from the UWO Directory. 8. Conversation, January 1995. 4 WordPerfect Office) running on Novell servers. Many UWO faculty use the hospitals as their ``home'' e-mail base. Note: The local LAN-based model and the POP3-based model can be supported by the current client portion of at least Pegasus and newer versions of Microsoft Mail (aka Microsoft Exchange). Thus the choice of client doesn't always dictate the mail delivery model. Increasingly, previously LAN-only clients can make use of the often more flexible POP3-based service. 3.1.1 Usage We know that there are approximately 22 LAN-based e-mail servers on campus but due to the distributed nature of this approach it is difficult to determine the volume of use. While we don't have a count of the number of messages sent within such systems we do know that during a typical day (at the end of November 1995) about 14% of the messages that pass through the ITS central mail servers originate on LAN-based systems. If we look at the UWO Directory we find that of the 3301 addresses currently registered there, approximately 15% have listed addresses from LAN-based servers. 3.1.2 Problems The simple SMTP gateways provided by these products rely on a smart SMTP gateway to deliver most e-mail. This can present a heavy load(9) on the central server(s) that provide this service (julian.uwo.ca in our case). LAN servers do handle client interactions that otherwise would have to be provided by a central POP3 server (currently also provided on julian) so there are advantages to the distributed approach as well. One difficulty with most LAN-based mail systems is the complexity of accessing them via dial-up connections or from dumb terminal connections at conferences. Often a Citrix or PC-Anywhere-style of connection (using 2 PCs) is necessary (and often not available). A second problem is that most of these servers are currently managed quite independently of others --- which leads either to the requirement for lots of (expensive) expertise at many sites or, more often, poorly managed and sometimes unreliable mail services. 3.2 POP3 Servers Post Office Protocol version 3 or POP3-based servers use a simple, well defined Internet protocol (described in ------------------------- 9. During November 1995 we were averaging a message every 3 seconds, 24 hours a day -- up from one every 6 seconds in January 1995. 5 RFC1081(10)). Essentially a client program provides a user interface and some local storage while the Post Office computer receives and transmits messages when asked. The Post Office is located on a machine that is available 24 hours a day, while the client is only connected to retrieve mail. Typically, current client programs converse directly via an SMTP (Simple Mail Transport Protocol) server like sendmail on a Unix machine to transmit messages. The primary POP3-servers are popper from Qualcomm(11) for Unix systems, the mercury gateway that runs on Novell servers and the VAX/VMS-based POP3 server from TGV (used at SSCL, but not ITS because our VMS mail software predates TGV). Lately, commercial products like Microsoft Exchange are appearing that support POP3 clients directly. 3.2.1 IMAP Servers IMAP servers, which can be considered a variant of the POP3-style of server, extends POP3 to allow more of the work to be done on the server. All mail and folders for IMAP users remain on the server and there are many more message manipulation functions available to client programs that use this server. This is essential for people that need to access their mail and mail folders from more than one system (e.g. from on-campus as well as from a home-based machine). In addition, IMAP provides a better level of security for sending messages since IMAP clients use the IMAP server (where the user has been authenticated) to deliver mail messages. Most POP3 implementations, in contrast, don't have message-level control of message deleting and rely on un-authenticated SMTP connections to send mail. Public domain and commercial IMAP servers are available for Unix and VMS systems. SSCL is currently running an IMAP server (TGV's) on their VMS systems. 3.2.2 POP3-based clients Currently at UWO the primary client is Eudora for Windows and for Macs. A number of sites that use the Pegasus client in the LAN environment use a POP3-version of the Pegasus client communicating to a Mercury SMTP/POP3 gateway on a Novell server or to a Unix POP3 server. With the recent popularity of Netscape and the development of a built-in POP3-based client for that package in version 2, its use as a mail client has increased. Netscape's client handles embedded hypertext links and graphics better than most clients but current versions don't offer much in the way of sorting and filing of mail messages. All of the common POP3 clients support the MIME (Multipurpose ------------------------- 10. URL: http://www.cis.ohio-state.edu/htbin/rfc/rfc1081.html 11. Makers of both the public domain and commercial versions of Eudora. 6 Internet Mail Extensions) standard for encapsulating binary attachments. Pine and some of the host-based mail clients can also make use of IMAP or POP3-based servers. These are only in use at SSCL. 3.2.3 Usage Of the e-mail that passes through the main campus mail hubs, about 32% originates on POP3-based clients. As of November 1995 we were typically seeing over 1500 different POP3 users during a typical day. About 88% of these originated via modem connections. Of the approximately 10,000 registered users of pan- ther/mustang, almost 60% of them use the system during an average week and about 30% make use of the POP3 service. (About 40% of users are inactive!) 3.2.4 Problems A major problem with the current POP3-based service is that it assumes that users will use only a single PC to read and file their mail. If this doesn't hold, as when someone has both a home machine and a office machine, then there is no easy way to file a message at work and then access it from the home machine.(12) The design of the system also often pushes users into leaving all their mail on the server and only down-loading copies to their PCs. This allows a Pine session to get at their messages from any dumb terminal connection but can also lead to excessive disk usage as no messages are ever deleted from the server. An IMAP replacement for POP3 addresses this problem by keeping the messages on the server so that any machine can access them as well as programs like Pine from a simple terminal session. But it comes with a price: Unless the IMAP client caches e-mail, it means that off-line reading of mail is impossible. 3.3 Host-Based E-mail These are typically mail systems that run on large timesharing hosts. They communicate locally via the native file system and use sendmail or sendmail-like programs to communicate via SMTP to the rest of the Internet. The interfaces range from simple line-oriented, to full-screen and to X-window-system based applications. Unix: Pine, mh, xmh, mail, xmail, mailtool, elm ------------------------- 12. A connection to the office machine's file server is a possibility as is a PC-Anywhere or Citrix style of connection. 7 Cyber: X.400-based mail system (single host system) VMS: Unix mail port (ITS), vendor with TGV enhancements (SSCL), Pine (SSCL) MVS: Emc2/TAO (also a client/server version) Of the above list, all are supported by ITS or SSCL with the exception of xmail, mailtool and elm. The UWO Library Systems Group provides some support for elm on the Sun platform. Pine and mh support binary document exchange using the MIME format. 3.3.1 Usage By far, the most popular of the host-based e-mail systems is Pine. About 80 to 90% of users that log into the ITS Unix systems use Pine They generate over 3,500 messages per day. But we know that many Eudora users also use Pine so it is difficult to determine the numbers that use Pine exclusively. According to the UWO directory, about 82% of faculty and staff list addresses on machines with host-based mail systems. Of these, 56% use Unix or VAX-based systems that also support POP3 clients. 25% are registered with Emc2/TAO addresses. 3.4 Proprietary client/server-based tools The client-server version of Emc2/TAO and the VAX/VMS Pathworks mail system in use at SSCL are both examples of E-mail programs that, while following a model similar to POP3 or IMAP-based services, have been developed using private and proprietary interfaces and protocols. This means that the vendor's server must be used with the vendor's client program. Typically there is a high degree of integration and blurring between the client and the server in such systems. 3.4.1 Pathworks The Pathworks system supported by SSCL currently has a large number of users in the Faculty of Social Science but the Lab's stated(13) direction is toward IMAP and POP3-based clients using standard servers. We don't have a lot of experience with Pathworks. Our assumption is that either SSCL shields us well or (more likely) it works well. 3.4.2 Emc2/TAO Emc2/TAO clients: ------------------------- 13. Conversation with Les Flodrowski, January 1995. 8 o communicate with the local proprietary database-oriented mail system running under the MVS operating system o transport their data primarily via Novell IPX protocols but IP is an option that is used for about half of the connections o don't support MIME for the transport of wordprocessing documents or other complex data types o require the use a Citrix server port or revert to a dumb terminal connection to provide remote service o require more memory than is available on many workstations especially when main-line tasks need to be accomplished at the same time (but this criticism could be leveled at almost any Windows-based application!). The Emc2/TAO server o consists of at least three separate components (SMTP gateway to LU6.2 emulation, MVS for database, IP/IPX server gateway) which makes it difficult to track down problems (lots of finger pointing) o requires that an external SMTP gateway (SCO Unix-based) be available to allow communication with the rest of the world. This gateway seems to be unreliable and has to be restarted frequently resulting in long e-mail delivery times. It also has some conformance problems: -- can't produce conforming headers like the date field (and the vendor can't or is unwilling to help) -- uses the envelope from rather than the header From: for delivery -- doesn't scale well to large volumes -- uses a LU6.2 terminal emulation connection running on a small PC to connect to the host -- can't transport plain text reliably (truncates lines, mis-translates characters) which means that uuencoded messages (for example) cannot be sent or received successfully (never mind MIME!) -- re-writes non-local and local addresses that may be very difficult to move to a new system (due to length) i.e. peter.marshall@uwoadmin.uwo.ca even though he doesn't have an account on the machine uwoadmin.uwo.ca. 9 o has been implemented to use a private directory (that is difficult to keep synchronized with the main directory) and has no connection to the new student directory. o provides an e-mail to FAX gateway (but only for users of the package -- there's no gateway into that service no FAX receiving service) o provides calendaring and bulletin board (not integrated into the Usenet news system) services o is not accessible from Mac systems nor from terminal clusters at conferences due to a requirement that the Encore-based challenge/response system be navigated before logging into the MVS system. (Should a large user-base e-mail system be on the highly secure corporate-data machine?) o doesn't produce (or make it easy to produce) good usage summaries via a complete audit trail 4 Recommendations -- Principles Or lessons that we've learned. o Go the way of the Internet (SMTP/MIME), not way of the telcos (X.400), not proprietary solutions like Emc2/TAO or MAPI. o We must contend with and accommodate diversity especially in the choice of client software. o Add-on services (like FAX) must be made available to the whole community, not just a subset using a specific suite of programs. o E-mail should be available to all computer users. It should be available in their native environment. o E-mail and supporting folders of old mail should be available and accessible from anywhere (office, labs, home and conferences) without any special effort on the user's part. o E-mail doesn't do everything! There are other tools that are better for mass distribution of material (bulletin boards like the Web) and for group discussions (Usenet news and other conferencing software), for forms handling and for calendaring functions. Keep it simple. o There is a certain set of fundamental features that will be required in any e-mail system. Over time, this suite will grow as the base-level user gets more sophisticated 10 and volumes increase. Here's a list of such features: -- folders -- sorting and filtering capabilities -- Internet compliance -- ability to transmit binary files -- new mail notification -- simple, easy access to directory information -- an easy way to set an auto-response (vacation) message 5 Recommendations for the near term There are two areas for near term recommendations: 1. those that deal with simplifying and reducing the numbers of clients and servers supported on the campus network for e-mail and related communications services, and 2. those that involve an extension or enhancement of our existing service. 5.1 Simplification To be considered: o Drop support for DOS-based POP3 clients like NUPop. People run Windows. o Investigate the possibility of a campus license for a commercial e-mail package like Eudora Pro, cc:Mail, Z-mail, MS/Mail or even Lotus Notes(14) to provide more functionality than what is provided in freeware or shareware packages. Do these packages work within the IMAP client-server model? Are they open? Do they interact well with the Internet and with PeopleSoft-based systems? Can they be implemented widely at a reasonable cost? Are they worth that cost? o Investigate and consider supporting Netscape's bundled mail system. Is it IMAP capable? ------------------------- 14. But this package, from the outside, seems to be very closed and proprietary although Lotus is working on opening things up. 11 o Separately investigate central calendaring systems (proba- bly web-based perhaps even Java-based?). o Investigate using IMAP servers instead of POP3 servers to get the superior communicate-anywhere advantage that IMAP provides. o Encourage the phase-out of LAN-based email systems like Pegasus/Mercury in favour of large POP3 or IMAP based servers. o Make account setup and tear-down for faculty, staff and student e-mail accounts simple and, as far as possible, automatic. o To keep the diversity of supported products under control, do not provide formal support for Pegasus POP3-clients, but do not restrict its use or any other POP3-based client. o Immediately start on a migration of the existing ITS VAX (and Cyber NVE) mail users to other platforms. o Move users away from Emc2/TAO to a new ITS supported mail package. 5.2 Extensions To be considered: o As we move into an era where electronic mail is a basic service for everyone at the University (faculty, staff and students) we need to find ways to inexpensively handle very large numbers of mail users. o Automated support for departmental and other group mailing lists including private, moderated and public lists. (But encourage the use of Usenet news as appropriate.) o For power users find and implement tools for filtering mail either on the server or on the client system to automatically answer, file, sort or otherwise process the incoming deluge. See also the recommendation for commercial e-mail packages above. o Support for encryption and electronic signatures via Pretty Good Privacy (PGP) tools and procedures. 6 The Future We see the personal communication facilities characterized by e-mail today evolving into a multi-media exchange and collaboration system. The evolution will be slow and based on standards widely adopted around the Internet. 12 Voice, video, hypermedia-links will become commonplace in interpersonal communications. Filing systems must become more automatic and easier to search. World-wide directories (probably not based on X.500) will be integrated into the suites of programs that people use for communicating. POP3 will be superseded by protocols more powerful than IMAP to allow a better and more flexible split between the workstation and the server for mail functions. The distinctions among the clients for e-mail, web-browsing, news reading will continue to blur, as we are already seeing in a crude way with Netscape. Remote access with local functionality and full access to archives, databases and other related functions from anywhere will be required and expected from tools that will help us to collaborate more effectively. The area will continue to be volatile with some campus groups being much further ahead than others. The only way to bring some semblance of order and to permit universal interoperability is to implement standards that are widely supported and adopted by our peers in the academic community. These standards will continue to evolve. 7 Next Steps --- Back to Reality It is clear that we need to move toward the next generation of mail programs if only to handle the immediate needs of 1. the current users of Emc2, 2. the current users of ITS's VAX mail system and 3. the transition of users from the character-based Pine system toward a GUI-based interface. The first two are critical as the machines that support these applications are disappearing and, in the case of Emc2, ITS no longer has experienced server support people. The third can proceed at a pace comfortable to the users. One reason to accelerate a move from Pine might be to provide the ``universal'' mail service via a client-server (IMAP) model since that model is, in some ways, easier to control than a system that requires a direct login to a general purpose computer. To this end we suggest that a working group be established to determine in detail what enhancements to the campus electronic mail infrastructure will be required to accommodate the needs of users while providing continuity with the existing world-wide e-mail system. Since there are major server and client issues that need to be addressed and coordinated this must be a cross-group team. The group should include as a minimum: o Peter Marshall (Project Leader, Server Support, information and communications server specialist) 13 o Colleen Bretzlaff (Pine, VMS, Postmaster experience, information and communications server specialist) o someone with Unix applications and server support experience o someone with with MS-Windows applications (especially e-mail support) experience o someone with MAC applications (especially e-mail support) experience o someone experienced with Emc2 user support o someone doing LAN support and familiar with the common LAN-based e-mail packages 14