IOA-IPAD

The following is a list of the IPAD 2.0 new features which apply to mail service and processing.

• Mailing List Manager. A mailing list manager is now available to operate medium size mailing lists. Remember that the IPAD is your router, and if you make a lot of very large mailing lists, the IPAD has to generate all of these files and I feel certain there will come a point where performance starts to be severely impacted. So when you reach that point, you've reached the point where this mailing list handler isn't what you can use any more. It isn't designed to do large 5,000+ user mailing lists, so don't expect it to. It will do much larger mailing lists than TIGER did, and much faster and better.

This mailing list manager has been designed to allow control and operate like the most commonly used mailing lists on the Internet today, namely Listserv and Majordomo. While it only has a subset of their features (specifically that subset which is possible when you don't store an archive of your list messages locally), what it has should operate as users of those types of lists expect.

In addition, it has configuration to allow adaptation to the future trends in mailing lists such as special mailing list routing options in the header. Since these aren't fully known now, a general method has been implemented to handle them as they occur. Examples will follow:

Defining a mailing list for use with the IPAD

There is NO IPAD.CTL configuration to use mailing lists. The only configuration occurs in the MAIL_AUTH file, using the new LIST and LISTMNGR commands.

You define a mailing list using the LIST command as follows:

LIST <address> <ctl_file>

<address> is the email address of the list (e.g., ipadowners@example.com). <ctl_file> is the fully qualified name of the file that contains the configuration information for the list. The contents of this file define the attributes of the list as described below.

You define the access to the list manager (where email is sent to subscribe, unsubscribe, etc.) as follows:

LISTMGR <address> LIST=<ctl_file>

• <address> is the email address of the list manager access (e.g., sales-request@example.com)
• <ctl_file> is the control file for the list.

User Requests to a List Manager

Users have three control options through the list manager:

• 1. Request to subscribe to the list.
• 2. Request to unsubscribe from the list.
• 3. Request help on the list name.

Messages addressed to a LISTMGR address are processed by looking for one of the control verbs in either the Subject: field or the body of the message. Note: You can define the text for these control verbs in the LIST configuration file as described below.

List Manager for a Group Of Lists

An alternate form of the LISTMGR command allows a general manager entry point that can manage multiple lists. This form is:

LISTMGR <address> GROUP=<grp_file>

<address> is the email address of the list manager access (e.g., listserv@example.com)
<grp_file> is a file defining the lists which may be managed and contains lines of the form:

LIST=<address> <ctl_file>

which specifies the address of the list and its list control file. Note: A Group control file also accepts the commands:

SUBSCRIBE=
UNSUBSCRIBE=
HELP=

As for a list control file to define the action verbs as you wish.

It also accepts the commands:

HELP_FILE=
INVALID_FILE=

To define the response to a help request with no list named or a request with no invalid inputs.

A group file enables multiple lists to be managed by a single list manager. Note that in this mode, several list commands can be handled in a single message (no more than one per line). Commands are the manager command verb followed by the list address that it should apply to. For example:

Two lists are defined as:

1. a@b.com
2. b@b.com

The body of a message to a GROUP manager could say:

subscribe a@b.example.com
help b@b.example.com

And both requests would be honored.

Mailing List Control File

The mailing list <ctl_file> defines the behavior of a mailing list and contains records (one per line) of the form:

<verb>=<setting>

The commands and their meaning are:

ERRORS-TO=<address> Defines the address where errors should be sent (bounce messages, etc.) This should be an address a human monitors so that the list can be properly updated when errors occur.

REPLY-TO=<address> The address where standard replies should go. Generally this will be the address of the list, but it could also be a moderator address.

NAME=list name A text string that is the name of the list. If it is more than one word, it should be enclosed in quotes.

HEADER=<hdrline> An RFC-822 header line that should be replaced in the original message if present, or added to the message if not present.

Example:

Will place the X-Mailer line in the header of each list message sent out, while deleting any X-Mailer: line in the received message.

TYPE=<keyword>

• TYPE=RESTRICTED creates a restricted list (like the ipadowners@example.com list) which is a list where the received email that is not from a person in the list is ignored. Also no one can subscribe to the list.
• TYPE=ACCEPT_ALL creates a list which will accept and reflect all messages, including messages from people that are not subscribed to the list (TIGER defaults to this form).
• TYPE=MEMBERS creates a list which will only accept messages from addresses which are in the list, but which allows anyone to subscribe or unsubscribe. This is the IPAD default if no TYPE= verb is given.

ADDRESSES=<filename>

<filename> is the fully qualified name of the file where the addresses of list members will be found. Each address is listed on a separate line.

Note: Any address listed with a + as the first character is ignored.

If the list is TYPE=RESTRICTED, then names with a + are not in the list, but ARE allowed to subscribe. In a restricted list, an unsubscribe adds a + to a name rather than delete it, so a member can unsubscribe for a while and then re-subscribe.

If the list is TYPE=MEMBERS or TYPE=ACCEPT_ALL, then names with a + are not in the list AND they are NOT ALLOWED TO SUBSCRIBE.

A line with the word CLOSED (without quotes) is special in the addresses file. If a CLOSED line is found in the file, those members ABOVE the line are considered full members, while those below the line are not. A message sent to the list is NOT sent to members BELOW the CLOSED line UNLESS it originated from one of the full members above the line. Messages from members BELOW the CLOSED line WILL be sent to all full members ABOVE the CLOSED line but no on else. In effect, names above the closed line are moderators of the list. Note: The effect of CLOSED is the same regardless of list TYPE=.

HELP_FILE=<filename> <filename> is the fully qualified name of an ASCII text file to which is sent as a response to a LISTMGR help request message.

WELCOME_FILE=<filename> <filename> is the fully qualified name of an ASCII text file which is sent as a response welcoming new subscribers to a mailing list.

REMOVE_FILE=<filename> <filename> is the fully qualified name of an ASCII text file which is sent as a response when a person successfully removes themselves from a mailing list.

INVALID_FILE=<filename> <filename> is the fully qualified name of file to send when a person sends a LISTMGR a request that doesn't have any known keywords in it. A LISTMGR keyword can appear anywhere in the Subject: line or the body of the message, but no valid one appeared there so this file is sent as a response.

SUBSCRIBE=<keyword> <keyword> defines a valid string in a submitted control message which means the user wants to subscribe to a list. This command allows you to defined this keyword. The SUBSCRIBE command may appear as many times as you wish, to define synonyms for subscribe.

UNSUBSCRIBE=<keyword> <keyword> defines a valid string in a submitted control message which means the user wants to unsubscribe from a list. This command allows you to defined this keyword. The UNSUBSCRIBE command may appear as many times as you wish, to define synonyms for unsubscribe.

HELP=<keyword> <keyword> defines a valid string in a submitted control message which means the user wants help information for a list. This command allows you to defined this keyword. The HELP command may appear as many times as you wish, to define synonyms for help.

Sample Mailing List Configuration

The following example defines the IPAD owners mailing list. The list is to respond to the email address:

MAIL_AUTH Configuration:

LIST ipadowners@example.com ipadown.ctl
LISTMGR ipadowners-request@example.com ipadown.ctl

ipadown.ctl Configuration:

ERRORS-TO=list.keeper@example.com
REPLY-TO=ipadowners@example.com
NAME=IPAD Owners Mailing List
TYPE=RESTRICTED
ADDRESSES=ipadown.lst
HELP_FILE=ipadown.hlp
WELCOME_FILE=ipadown.wel
INVALID_FILE=ipadown.inv
REMOVE_FILE=ipadown.rem
SUBSCRIBE=subscribe
SUBSCRIBE=join
SUBSCRIBE=add
UNSUBSCRIBE=unsubscribe
UNSUBSCRIBE=delete
UNSUBSCRIBE=remove
HELP=help

ipadown.lst contents:

bhartman@example.com
phil@example.net
<other addresses, one per line>

• The verb ALLOW=<addr> has been added to mailing list control files to define email addresses and/or domains which are MEMBERS of this mailing list. You can have as many ALLOW=<addr> lines as you wish.

Only members may issue a subscribe request for the list. By default (if no ALLOW= verbs are present in a list control file) the membership is ALLOW=*@* (everyone is a member).

Wildcards here operate with the same rules as in the MAIL_AUTH file. If a non-member attempts to subscribe (i.e., their email address does not match an ALLOW= statement in the list control file) the subscribe request will be rejected.

Notes: The interaction of ALLOW= and TYPE= on list actions is:

• ALLOW= verbs are ignored in TYPE=RESTRICTED lists since a RESTRICTED list's membership is already defined by the email addresses present in the ADDRESSES= list file (with or without a leading + indicating currently inactive).
• ALLOW= is particularly useful for TYPE=MEMBERS lists, as it restricts the potential membership to the list. In essence, a TYPE=MEMBERS with ALLOW= restrictions on the membership makes a semi-restricted list which is open, but only to certain email addresses.
• ALLOW= with TYPE=ACCEPT_ALL lists restricts who can subscribe to the list, but will still echo messages from non-members to all members who have subscribed (i.e., anyone can mail to the list, but only member will receive the mail).

Example:

TYPE=MEMBERS

ALLOW=*@example.com

This would allow bhartman@example.com to subscribe, but would not allow bhartman@lab.example.com to subscribe.

ALLOW=*@example.org
ALLOW=*@*.example.org
ALLOW=*@example.net
ALLOW=*@extra.example.net
ALLOW=phil@example.com

This would allow anyone in domain example.org (including sub-domains), anyone at top level example.net or sub-level extra.example.net and the single address phil@example.com to subscribe.

• Added STRIP=<header tag> in mailing lists. <header tag> is any header line (including the colon) which should be stripped from incoming messages. This allows the list supervisor to strip out any lines which are deemed inappropriate for a mailing list. A partial listing should almost certainly include:

STRIP=Return-receipt-to:
STRIP=Priority:
STRIP=Precedence:
STRIP=X-MSMail-Priority:
STRIP=X-Priority:
STRIP=X-Sender:
STRIP=X-MimeOle

These are items that have been seen in the ipadowners@example.com mailing list and probably are inappropriate for general distribution. Certainly the Return-receipt-to: line is inappropriate as Steve Sherwick will attest!

Note, as is the case with HEADER=, the following header fields cannot be stripped or modified:

Approved:
From:
To:
Date:
Message-Id:
Subject:
Received:
Sender:
Reply-To:
Errors-To:

While on the subject, it probably also makes sense to have a minimum number of header lines that get exchanged out. These should include:

HEADER=X-Mailer: IPAD 2.0
HEADER=X-List-Help: <mailto:listserver@here.com?Subject=help>
HEADER=X-List-Subscribe: <mailto:listserver@here.com?Subject=subscribe>
HEADER=X-List-Unsubscribe: <mailto:listserver@here.com?Subject=unsubscribe>
HEADER=X-List-Post: <mailto:listname@here.com>

Replace listserver@example.com and listname@example.com with the things that are appropriate for your list. The last four header lines implement what is currently a draft specification for mailing list header lines that make it easier for mere mortal end users to use mailing lists. The theory is that someone can click on a URL to do various functions with the mailing list (the four functions shown in the case of an IPAD). There will probably be mail clients out in the future which will put buttons on the toolbar for these functions if the proper header lines are found in the message (David Harris, author of Pegasus Mail has pledged to implement some sort of client side implementation as an example).

• Mail from the IPAD outbound directory to a POP3 account on the IPAD will now skip the SMTP session stuff and just go directly to adding the message to the POP3 folder. This should significantly reduce IPAD overhead for mailing list messages that go to recipients with POP3 accounts on the same IPAD. This will show in the client/server window as Internal POP mail.
• In mailing lists, if REPLY_TO=NONE is present in the control file, then the Reply-to: field will now be stripped from the outgoing messages. This allows mailing lists where replies go to the individuals that posted the message, rather than the list itself. This can be thought of as a one way mailing list. When used in conjunction with a CLOSED (moderated list), such as the announce@example.com mailing list, this allows a select number of individuals to post to the list, and for them to receive the replies directly.
• A validate address check on incoming SMTP mail has been added which will reject mail addresses as a syntax error if illegal characters are used, or if the address has multiple @'s in it. It also will reject mail that has a top level domain name (last two portions of domain part) which does not follow the Internic rules of having only letters, numbers and hyphens in the name. This should detect and reject a small amount of incoming spam now that spammers are using invalid domain names.
• While a mailing list is generating messages, MLST msg generation will appear in the client/server portion of the console screen.
• The use of the MAIL_AUTH file has been changed to dramatically speed up email and lower its impact on the IPAD. The MAIL_AUTH file is now compiled to a temp file with a RAM index, and the actual MAIL_AUTH file is not scanned. If the MAIL_AUTH file is changed (seen by the IPAD as a change in its date/time from the compiled version) then the new file will be dynamically re-compiled on the next email access.

This change has the following good effects:

1. It makes looking things up in MAIL_AUTH MUCH faster than before. The result is that REJ*, DIS*, BAD* records no longer impact system email performance very much at all and POP3 login is extremely rapid regardless of the order or size of the MAIL_AUTH file.

2. On systems that are very busy with email, updating the MAIL_AUTH file is now not only possible but easy. The update should be accomplished via FTP or CIFS as follows:

a. Copy in the new MAIL_AUTH file under a temporary name.

b. Delete the current MAIL_AUTH file.

c. Rename the new MAIL_AUTH file to the real name.

Note: The indexing of MAIL_AUTH takes approximately 25K of memory for each 500 records. This makes it even more important that older IPADs have at least 8MB of RAM installed. The first MAIL_AUTH lookup will take slightly longer than before because the MAIL_AUTH file has to be compiled into a temp file. After that, lookups are DRAMATICALLY faster.

Note: If you delete MAIL_AUTH, the IPAD continues to use the last one it compiled until a new MAIL_AUTH file shows up. This is why the above update method works cleanly.

• Implemented the ETRN command in ESMTP to allow the remote to request an immediate scan for any outbound email. RFC1985 minimum implementation was done, seems to be all that should be required.
• Implemented a new MAIL_AUTH verb COPY which will generate a single copy of a message that is received and remail it. The command has the same syntax as REDIR which is:

COPY <name> <copyto>

<name> and <copyto> can have wild cards with the same rules as REDIR. The copy is placed in the IPAD outbound mail directory and sent via SMTP so it may go anywhere, not just to local POP boxes or inbound UUCP gateway directories.

NOTE: COPY will only make a single COPY, and it will only trigger on the TO: name of the received message. That is, a COPY will not be activated by the target address of a REDIR verb (this is the same rule described in detail below for pass-thru responders).

Example:

POP3 pbecker@example.com pbecker pswd c:\pop3\pbecker
COPY pbecker@example.com phil@example.net

Any message received by pbecker@example.com is placed in the POP box and a copy of it is generated and mailed to phil@example.net.

Note: If you generate a COPY loop, it will make 5 copies and then stop with a cannot route email error message just like any other routing loop does.

• Implemented a new option on the RESP verb for the MAIL_AUTH file to create a PASS_THRU responder instead of the normal DEAD_END responder. The syntax of the RESP command is now:

RESP <name> [d:[\path\]file[.ext]] [PASS_THRU | DEAD_END]

The default option is DEAD_END if neither is specified, for backward compatibility. If PASS_THRU is chosen, the responder is no longer a destination for the message, but rather becomes an acknowledgment of the message's handling. The response file is sent out as a message the same as with a DEAD_END responder, but rather than discard the message and end processing, the message is routed as if the RESP command didn't exist. This allows several new capabilities for example:

Keeping the messages sent to an auto-responder:

RESP info@example.com c:\text\info.txt PASS_THRU
POP3 info@example.com info pswd c:\pop3\info

This same structure becomes a notification mechanism as well:

RESP tech-support@example.com c:\text\techgot.txt PASS_THRU
POP3 tech-support@example.com tech pswd c:\pop3\tech

Using REDIR with PASS_THRU responders

The introduction of pass-thru responders creates many interesting ways to tie things in knots when combined with REDIR, and I've made some rules I think are easy to understand to resolve these situations. Let me first create an example using a sample MAIL_AUTH file:

RESP sales@example.com c:\text\thankyou.txt PASS_THRU
REDIR sales@example.com jgreen@example.com
RESP jgreen@example.com c:\text\vacation.txt PASS_THRU
REDIR jgreen@example.com pbecker@example.com
RESP pbecker@example.com c:\text\philgot.txt PASS_THRU
POP3 pbecker@example.com pbecker pswd c:\pop3\pbecker

Example: A message is sent to sales@example.com.

The desired result is for the sender to receive the thankyou.txt file which tells him he will get a response from a salesman and maybe gives him directions to the web site, to info@example.com for lots of info, etc.

Then his message should be redirected to Joe Green's mailbox for handling.

However, Joe has gone on vacation, and has put a responder on his mailbox to say he'll be back next week and his mail is being handled by me. And he has re-directed his mail to me.

The problem: The user who sent mail to sales@example.com wouldn't understand hearing that Joe was on vacation, he just needs to know that someone will call as per the normal sales responder. But if someone sends mail directly to jgreen@example.com, they WILL want to know that Joe is on vacation, since it may be part of some discussion I don't know about and can't handle until Joe returns. And in no case do they want the response that I got the message and will be back to them within 24 hours!

To address these issues I've made ONE rule about pass_thru responders and redirection as follows:

1. A PASS_THRU responder will ONLY trigger if it matches the address of the arriving message. No PASS_THRU responder will trigger as the result of a REDIR routing change.

Note that DEAD_END responders will trigger whenever they are hit, via a REDIR or not as always. They don't create these problems. I think that this rule will make PASS_THRU responders tame enough to allow them into the world at large and not cause mail storms of multiple responses to a single message.