NELUG, 7/6/2000
Eddy Younger (eddy@shofar.uklinux.net)
Software Components of the Email System
There are principally three classes of software components involved in transfering a mail message from the sender to the recipient:
- MUA – mail user agent, used to read, compose and post mail.
- MTA – mail transport agent. MTA’s at the source and destination hosts (and possibly also intermediate hosts) pass the messages from one to another
- MDA – mail delivery agent. At the ultimate destination host, the MDA receives the message from the local MTA and delivers it to its ultimate destination, usually the recipient’s mailbox file.
In the modern world almost all email transport is achieved using SMTP – the Simple Mail Transport Protocol – or its Extended variant ESMTP. MTA’s speak to each other in (E)SMTP. You can send mail without using a MUA if you wanted to, by talking SMTP directly to the MTA, and it used to be possible to do all kinds of nefarious things by doing so, though thankfully most mail servers are much more secure in these days of Skript Kiddies and Spam.
MUA’s
Reading Mail:
Commonly there are three methods to access the mailbox:
- read the mailbox directly (/bin/mail, most Unix mail clients);
- POP3 – Post-Office Protocol. Mail reader communicates with POP3 server which accesses the mailbox and transfers messages to the mail reader, which re-files them locally;
- IMAP – another client/server system. The server is capable of transfering header information to the client without the message body, and of deleting messages from the mailbox on behalf of the client. The messages need only be uploaded to be read or filed.
Modern Unix mail clients (pine, mutt, emacs VM) can handle POP, IMAP or a local mailbox. Others such as Netscape need to talk to a POP or IMAP server, even if the mailbox is local.
Sending mail:
MUA hands mail on to a MTA. This may be on the local host or on a remote host (maybe at your ISP), depending upon which MUA you’re using and how it’s configured. The MUA generally uses SMTP to talk to the MTA; in rarer (rarer on Unix, that is) case POP may be used to post mail as well as to read it.
MTA’s
Sendmail is the most widely used, most powerful, probably the most efficient, and (formerly) the hardest to configure. It originates from a time when memory and CPU bandwidth were expensive, and when there were many and varied networks other than the Internet (BITNET, EARN, JANET, UUCP …). Sendmail is capable of acting as a gateway between such different networks, as a mail-to-fax or mail-to usenet gateway, etc. It is very resource-efficient. Virtually every piece of mail which moves on the backbone is handled by sendmail at some point (chalk up one more for Open Source Software).
Since the virtual demise of all other networks, numerous other MTA’s with more limited capabilities have come along, e.g Exim, qmail, Postfix. These tend to lack the gateway capabilities of sendmail, for example, but as a result are simpler, and regarded by some as more secure. A major selling point seems to be that their configuration files are more readable than sendmail’s, which is undoubtably true but largely irrelevant as we shall see. Some very large sites are now running on these alternative MTA’s however – Freeserve runs on Exim for example.
MDA’s
The role of the MDA is to accept a message from the MTA on the final destination host and deliver it to its destination, usually a user’s mailbox. On Unix systems (ATT variants) the MDA for mail to a user’s mailbox is usually /bin/mail (though interestingly the MDA in the default sendmail.cf on Slackware Linux is procmail). For mail to a program (where allowed) it is /bin/sh.
The obligatory diagram
This picture may be complicated by
- DNS MX records for the destination
- Aliases, either at the source or destination
MX records
Once the MTA has deciphered the destination address, it does a name-server lookup (usually DNS, but maybe NIS or whatever) to find it’s IP address. MX records are special DNS records used by the mail system: a destination (e.g durham.ac.uk) may one or more MX records associated with it which point to the IP adresses of hosts which will recieve mail on behalf of that destination. In such cases, the destination may be a host which is incapable of receiving mail (no MTA), or may not exist as a physical machine at all (e.g. durham.ac.uk or shofar.uklinux.net). The MX records for these destination point to a real machine to which mail will actually be routed (a destination may have several MX records, possibly with different preference values).
Aliases
There are two classes of aliases:
- system wide, defined in /etc/aliases or the mail.aliases nis map; these are expanded by the MTA
- user’s aliases, usually defined in ~/.mailrc; these are expanded by the MUA when the message is posted
Configuring sendmail
The main configuration file for sendmail, by default /etc/sendmail.cf, is large and difficult to read even in the simplest configuration. For this reason sendmail has a reputation for being hard to configure. However, nowadays sendmail.cf files may be generated using the m4 macro processor from a source file of typically less than a dozen lines by calling out the pre-defined macros which form part of the sendmail distribution. Directly editing sendmail.cf is only necessary in unusual configurations.
sendmail.cf files contain a set of definitions (which essentially set values for macros used elsewhere) and rewriting rules which rewrite the headers of mail messages – most particularly the address portions – before delivering them to the destination (or next) MTA. The rules consist of a LHS which is a pattern to match against the input, and a RHS which defines how a matching input should be re-written. A rule is applied recursively until the match fails (or it is explicitly terminated), and rules may “call” other rules. The “entry point” is rule 3, confusingly enough.
You can now configure sendmail without knowing or understanding anything about the syntax of sendmail.cf
Configuring email on a dial-up host
There are two basic approaches, with various possible points in between:
Local MUA only
In this scenario, there is no mail service at all on the local machine. A mail client is used which can download mail from the dial-up service and can hand over sent mail directly to the MTA on the server. This is the approach taken by ISP’s in the software they supply to end users. This will work fine if you don’t need anything sophisticated like multi-drop mailbox capabilities, or to provide mail services to a lan through a dial-up gateway (though even that may be possible up to a point). Given the capabilities of a Linux box it’s pretty uninspiring though.
A real email service
As usual, there’s more than one way of doing it. We have to cope with the fact that the connection from the server to the internet (the dial-up connection) comes and goes. We will also usually need to deal with the fact that
- we only have one mailbox at the ISP into which all mail for <any_user>@<ourname>.isp.co.uk is delivered.
- we have to collect email via POP or IMAP, and not SMTP through our local MTA.
What follows is a case study based on my own setup, where one machine acts as a mail hub and also a dial-up gateway for a local area network. Sending mail to the outside world, and collecting mail from two separate ISP’s, are dealt with separately owing to the constraints of ISP’s and dial-up links.
Sending mail
Sending of mail is dealt with by MTA’s (in this case, sendmail) running on the LAN clients and the mail hub. Mail clients (MUA’s) hand over mail to the MTA on the local machine (in the case of Netscape mail and friends they could hand it straight to the mail hub). Sendmail on the clients uses the dumbest possible configuration, known as the “nullclient” configuration after the m4 macro used to generate it:
include(`../m4/cf.m4') include the boilerplate
sendmail.cf content
VERSIONID(`@(#)generic-shofar.mc 8.9 (Berkeley)') gets included as a comment
in the .cf file
OSTYPE(linux)dnl tells m4 which definitions
to use for platform-specific
paths, etc
DOMAIN(generic)dnl include vanilla domain-
specific definitions
FEATURE(nullclient,moriah) hand over all mail to the
MTA on the machine called
"moriah"
m4 configuration file for dumb lan clients
This generates a sendmail.cf file of 642 lines. The only support this provides is for forwarding all mail to the named mail hub. No aliases are expanded on the originating host, nor are .forward files processed – this implies that user acounts should be visible on the mail hub if .forward files are to be used.
The sendmail configuration on the mail hub needs to be more complicated. It’s major job is to rewrite senders’ mail addresses to be that of the ISP mailbox, in order that mail can be replied-to, and also to prevent anti-spam and anti-spoofing rules from rejecting the message. Well-configured MTA’s will refuse to accept messages whose “From:” address does not resolve to a valid host or MX record – this presents a problem for hosts or LAN’s with only dial-up connections, which should be using network addresses from the defined set of private ranges, and whose local host/domain names are not resolvable out in the real world. The second (though optional) function of the mail hub’s configuration is to define a “smart host” within the ISP to which all outgoing mail is passed. This has the advantage that if the ultimate destination of a message is temporarilly unavailable, the smarthost and not our mail hub will take care of retrying the delivery. It also means that if the link to the destination is very slow we don’t have to stay connected until the delivery is complete – the smarthost will accept the mail and we can hang up the phone and allow it to take care of the delivery. The disadvantage is that if delivery fails we won’t find out until we next connect to dowload our mail. Note that some ISP’s (e.g. Freeserve) interpose a smarthost whether you ask for one or not.
include(`../m4/cf.m4')
VERSIONID(`@(#)mailhub-shofar.mc 8.8 (Berkeley)5/19/98')
OSTYPE(linux)dnl
DOMAIN(generic)dnl
define(SMART_HOST,smtp:mail.uklinux.net) define the smart host and the
mailer thru' which to access it
MASQUERADE_AS(shofar.uklinux.net)dnl From: and Reply-To: addresses must
be rewritten as this
MASQUERADE_DOMAIN(shofar.org.uk)dnl all addresses in this domain are to
be masqueraded as above
FEATURE(allmasquerade)dnl causes local aliases and To:
addresses in above domain to
be masqueraded
FEATURE(masquerade_entire_domain)dnl rewrite <any_host>.shofar.org.uk to
be shofar.uklinux.net
FEATURE(masquerade_envelope)dnl masquerade the envelope as well as
the From: line (see below)
MAILER(local)dnl mail will either be delivered to a
local MDA
MAILER(smtp)dnl or transferred via SMTP
m4 configuration file for dial-up mail hub
Most of the masquerade features in this configuration are the result of the fact that I am using a bogus domain name (shofar.org.uk) on the local LAN. We want to be able to send mail to aliases defined locally (e.g. NELUG-announce) and have that rewritten as <local_alias_name>@shofar.uklinux.net ( hence FEATURE(allmasquerade) ), so that recipients can “reply-to-all” and it will work (but probably not the way they expect). We also want mail from any host on the LAN to have its address masqueraded ( FEATURE(masquerade_entire_domain) ). Most importantly, we want the envelope From address (not just the From and Reply_To in the header) to be masqueraded, as this is the address used to
- return bounces; and
- check for spoofed sender-addresses
It is therefore imperative that the envelope From address is resolvable to a valid address, which “eddy@shofar.org.uk” definitely is not (outside my private LAN).
Thus, all mail originating in the LAN is passed to sendmail on the mail hub, who will attempt to transfer the messages to the smarthost. If this fails, either because the dialup link isn’t up or the smarthost is broken, sendmail places the messages in a queue (by default /var/spool/mqueue), and will retry each message at intervals (the -q commabd-line parameter), warn the sender after a specified (in sendmail.cf) period, and give up and return the mail if a time limit (you guessed…) is exceeded. Because it is more usual for the link to be down than up, we don’t want to rely on sendmail happening to retry at a time when the link is actually there. So, one thing that the ip-up script does (whenever the link comes up, remember…) is to run “sendmail -q” which causes an instance of sendmail to start, process all queued message once, then terminate. Provided the smarthost isn’t broken, this will cause the queue to be flushed.
Generating the .cf file
Simplicity itself. Assuming you’ve created the m4 file and named it “<hostname>.mc” – a good idea if you’re going to produce config. files for several hosts – in the cf/cf directory of the sendmail distribution, simply “make <hostname>.cf” in the same directory, and your half-dozen lines of m4 source will be translated into many hundred lines of mystic runes. Copy the resulting “host.cf” to /etc/sendmail.cf on the target machine, kill and restart sendmail if it’s running in daemon mode. No, you don’t need to reboot, and you don’t need to insert the distribution cd several times over.
Receiving mail
To receive mail, we want to poll the mail services at two different ISP’s, which support only POP for mail retrieval, and deliver the messages to the appropriate mailboxes on the local mail hub. Client machines on the LAN all mount the mail spool directory (/var/mail) via NFS and so have direct access to the mailboxes. To provide for users of Netscape mail (or Windoze mail clients) we’d need to run a POP server on the mail hub, but up with that I no longer have to put.
The ISP mailboxes are multi-drop, i.e. if I requested an email address of “eddy@shofar.uklinux.net”, then “<any_user_name_at_all>@shofar.uklinux.net”, is a valid email address, and the mail will end up in the same mailbox at the ISP.
To collect mail, we use Eric Raymond’s most excellent fetchmail program, which deals quite nicely with all these requirements. It can be configured to poll a number of mailboxes (using POP3 or IMAP), retrieve the mail and forward via sendmail to either a single local user, or to map user-names in the recipient address of the message on a one-to-one basis to users on the local system and forward mail accordingly. User names in the mail address need not be the same as those on the local mail hub – fetchmail takes care of the mapping.
Fetchmail configuration
Fetchmail is run from the ip-up script whenever the dial-up link comes up; it is configured to poll the mailboxes at 5 minute intervals while the link remains up. The configuration file is .fetchmailrc (in root’s home directory since fetchmail runs as root from ip-up). In order for it to run without intervention, the .fetchmailrc file contains passwords to log in to the POP3 accounts, which represents a security risk in some environments despite it being unreadable except by root.
# Configuration created 11th May 2000 by hand
#
set syslog #error logging via syslog
set postmaster "postmaster" #last-resort recipient if mappings fail
set no bouncemail #don't bounce mail to sender on error
#
# "Single-drop" ISP mailbox
#
poll pop.freeserve.net with proto POP3
user "shofar.freeserve.co.uk" there with password "**********"
is eddy here options flush
#
# "Multi-drop" ISP mailbox
#
poll mail.uklinux.net with proto POP3 aka shofar.uklinux.net
user "shofar" with password "**********" to eddy nelug=eddy NELUG-admin
nelug-approval=eddy NELUG-list root keith here options flush
.fetchmailrc for dial-up mail hub
The first mailbox (pop.freeserve.net) is a simple case where we want all mail to be delivered to a single user on the local machine, regardless of what the username in the mail address might be. In the second case, the “to” keyword defines this to be a multidrop mailbox, and we list recipient names corresponding to local user names: remote-name=local-name specifies a mapping from the name in the mail address to a local user name; names without a corresponding “=local-name” mean that this name in the mail address should be mapped to the same local user name (equivalent for example to “eddy=eddy”, “NELUG-admin=NELUG-admin” in the above). Any mail address containing a username which is not listed will be delivered to the username specified in the postmaster variable set in the preamble (in this case “postmaster”).
In the second case, fetchmail will actively parse the mail headers in order to determine who the mail was addressed to (unlike the first case where it doesn’t care, it just forwards everything to “eddy”). For this reason, we need to specify using the “aka” keyword the host part of the mail address we should expect to find, as this is different from the hostname of the machine we are retrieving the mail from.
Debugging
Sendmail (and fetchmail if you “set syslog”) log messages via syslog to say what their doing. Sendmail in particular logs messages about every mail message it processes.
Example:
Jun 6 18:13:42 moriah fetchmail[295]: POP3> TOP 3 99999999 Jun 6 18:13:42 moriah fetchmail[295]: POP3< +OK Top of message follows Jun 6 18:13:42 moriah fetchmail[295]: reading message 3 of 4 (3512 octets) Jun 6 18:13:42 moriah fetchmail[295]: SMTP> MAIL FROM:<ey86798@herring.uk.sun.c om> BODY=7BIT SIZE=3512 Jun 6 18:13:42 moriah fetchmail[295]: SMTP< 250 <ey86798@herring.uk.sun.com>... Sender ok Jun 6 18:13:42 moriah fetchmail[295]: SMTP> RCPT TO:<eddy@localhost> Jun 6 18:13:42 moriah fetchmail[295]: SMTP< 250 <eddy@localhost>... Recipient o k Jun 6 18:13:42 moriah fetchmail[295]: SMTP> DATA Jun 6 18:13:42 moriah fetchmail[295]: SMTP< 354 Enter mail, end with "." on a l ine by itself Jun 6 18:13:43 moriah fetchmail[295]: SMTP>. (EOM) Jun 6 18:13:43 moriah sendmail[341]: SAA00341: from=<ey86798@herring.uk.sun.com >, size=3530,, pri=33530, nrcpts=1, msgid=<14652.56766.948321.273973@her ring>, bodytype=7BIT, proto=ESMTP, relay=root@localhost [127.0.0.1] Jun 6 18:13:43 moriah fetchmail[295]: SMTP< 250 SAA00341 Message accepted for d elivery Jun 6 18:13:43 moriah fetchmail[295]: flushed Jun 6 18:13:43 moriah fetchmail[295]: POP3> DELE 3^M Jun 6 18:13:44 moriah fetchmail[295]: POP3< +OK Message deleted Jun 6 18:13:44 moriah sendmail[342]: SAA00341: to=<eddy@localhost>, delay=00:00 :02, xdelay=00:00:01, mailer=local, stat=Sent Jun 6 18:13:44 moriah sendmail[335]: SAA00334: to=eddy , delay=00:0 0:06, xdelay=00:00:05, mailer=local, stat=Sent
This shows fetchmail retrieving a message via POP3, forwarding it to sendmail for delivery, and subsequently deleting the message from the POP server. It also shows the messages logged by sendmail on receipt and delivery of the message (the two sets of messages are interleaved in the file – a not uncommon occurrence in log files).
Sendmail may also be run from the command line is debugging modes to allow one to see exactly how it’s processing addresses, for example to produce a trace of the rewrite rules as they fire, and to trace the rewriting of the address. See the bat book or the red book for details of debugging flags.
The mail headers themselves contain valuable information which can help debug bounces or mail forwarding loops. Each MTA adds its own “Received…” line as it processes the message. For sendmail, the format of the Received: line is defined in sendmail.cf.
Example:
From paulmidgley@freeuk.com Sat Jun 3 20:58:33 2000
Received: from localhost (root@localhost [127.0.0.1])
by moriah.shofar.org.uk (8.9.3/8.9.1) with ESMTP id UAA00617
for <eddy@localhost>; Sat, 3 Jun 2000 20:58:31 +0100
Received: from mail.uklinux.net
by localhost with POP3 (fetchmail-5.1.2)
for eddy@localhost (multi-drop); Sat, 03 Jun 2000 20:58:32 +0100 (BST)
Received: from scrabble.freeuk.net (scrabble.freeuk.net [212.126.144.6])
by www.uklinux.net (8.9.3/8.8.7) with ESMTP id SAA09791
for <eddy@shofar.uklinux.net>; Sat, 3 Jun 2000 18:26:30 +0100
Received: from [212.126.152.202] (helo=z4t4z8)
by scrabble.freeuk.net with smtp (Exim 3.12 #1)
id 12yHgW-00022v-00
for eddy@shofar.uklinux.net; Sat, 03 Jun 2000 18:26:08 +0100
Message-ID: <001301bfcd7f$fb61e220$ca987ed4@z4t4z8>
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Lookout Excess 5.00.2615.200
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2615.200
From: "Paul Midgley" <paulmidgley@freeuk.com>
To: "Eddy Younger" <eddy@shofar.uklinux.net>
Subject: Internet access
Date: Sat, 3 Jun 2000 18:18:25 +0100
Here we can see the route the message took from sender to recipient and the MTA’s involved at each stage. The message was generated by something called “Microsoft Outlook (??)”. It was transferred first of all to scrabble.freeuk.net which was running Exim 3.12. scrabble.freeuk.net passed the message to www.uklinux.net, which is running sendmail v8.9.3: this is despite the fact that the message was addressed to shofar.uklinux.net – this probably indicates that there is an MX record for shofar.uklinux.net pointing to www.uklinux.net. (The version numbers in parentheses are sendmail-version/config-file-version. config-file-version is actually the value of the macro Z defined in the config file). The message has now reached its destination according to the address in the envelope, but this is not the end of its travels, as fetchmail now comes along and removes it from the mailbox, rewrites its envelope address and re-inserts it into the email system by passing it to sendmail (v 8.9.3) on moriah.shofar.org.uk. As the envelope To address has been rewritten by fetchmail to be the local host, sendmail delivers it locally. It would be perfectly possible though for the envelope address written by fetchmail to be a local alias which expanded to a remote address, and hence for the message to be re-routed over the internet – this is how the old NELUG mailing list worked.
Example 2: the above config. in action
From owner-nelug-eddy=shofar.uklinux.net@lists.lug.org.uk Thu Jun 1 21:12:45 2000
Received: from localhost (root@localhost [127.0.0.1])
by moriah.shofar.org.uk (8.9.3/8.9.1) with ESMTP id VAA00390
for <eddy@localhost>; Thu, 1 Jun 2000 21:12:44 +0100
Received: from mail.uklinux.net
by localhost with POP3 (fetchmail-5.1.2)
for eddy@localhost (multi-drop); Thu, 01 Jun 2000 21:12:44 +0100 (BST)
Received: from lug.org.uk (qmailr@lug.org.uk [195.92.249.253])
by www.uklinux.net (8.9.3/8.8.7) with SMTP id VAA29981
for <eddy@shofar.uklinux.net>; Thu, 1 Jun 2000 21:24:38 +0100
Received: (qmail 13274 invoked by uid 300); 1 Jun 2000 20:24:19 -0000
Delivered-To: nelug@lists.lug.org.uk
Received: (qmail 13265 invoked from network); 1 Jun 2000 20:24:19 -0000
Received: from ns0.uklinux.net (HELO www.uklinux.net) (root@212.1.130.10)
by lug.org.uk with SMTP; 1 Jun 2000 20:24:19 -0000
Received: from moriah.shofar.org.uk (root@ppp-1-201.cvx1.telinco.net [212.1.136.201])
by www.uklinux.net (8.9.3/8.8.7) with ESMTP id VAA29961;
Thu, 1 Jun 2000 21:24:13 +0100
Received: (from eddy@localhost)
by moriah.shofar.org.uk (8.9.3/8.9.1) id UAA00333
for NELUG-announce; Thu, 1 Jun 2000 20:55:21 +0100
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Message-ID: <14646.40424.81455.594113@moriah.shofar.org.uk>
X-Mailer: VM 6.72 under 21.1 (patch 8) "Bryce Canyon" XEmacs Lucid
Precedence: bulk
Reply-To: nelug@lists.lug.org.uk
From: Eddy Younger <eddy@shofar.uklinux.net>
Sender: owner-nelug@lists.lug.org.uk
To: NELUG-announce@shofar.uklinux.net
Subject: [nelug] Reminder - NELUG meeting, Weds. 7th June
Date: Thu, 1 Jun 2000 18:31:20 +0100 (BST)
Recommended Reading
- The Bat Book: “Sendmail” – Costales, Allman & Rickert, O’Reilly 1993.
- The red book: “Unix System Administration Handbook” – Nemeth, Snyder, Seebass & Hein, Prentice-Hall 1995
- The Crab Book; “TCP/IP Network Administration”, – Hunt, Craig, O’Reilly.
- cf/README file from the sendmail distribution;
- /usr/doc/fetchmail*/*, fetchmail man page.
- RFC 821 (SMTP definition); RFC 822 (mail mesage structure); RFC 1425 (ESMPT definition); RFC 974 (DNS MX records and mail routing)