#
#	$Id: README,v 1.7 2016/05/17 10:38:03 craski-shell_86 Exp $
#


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Function
# ========
#

OpenBSD pf spamd auto-whitelister of round-robin SMTP clients


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Description
# ===========
#

ungrey-robins assists postmasters by automatically whitelisting
round robin SMTP clients (which often fail to pass greylisting),
without resorting to manual maintenance of whitelists.


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Name
# ====
#

The project name of 'ungrey-robins' is a word play on:
  o greylisting
    o the 'un-' prefix, to mean not grey
  o round-robin
  o the popular and silly computer game 'Angry Birds'

ungrey-robins is a computer tool which ungreylists silly squawking round robin
SMTP sending mail servers, which stupidly play about with established Internet
mail delivery standards. These commercial companies are not running businesses
up to even the carrier pigeon grade.


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Dilemma
# =======
#

The rise of spam abusing Internet email users (largely from virus infected
zombie consumer devices), the most effective method available to postmasters to
not accept delivery is greylisting.

Rather than outright rejecting unknown email connections (blacklisting), a
designed robustness feature of the Simple Mail Transfer Protocol is for sending
clients to retry when the receiving server returns particular error codes.

Due to zombie virus senders urgency to deliver as much spam before being caught,
rarely do their authors adhere to Internet standards, and sending a vague retry
code either foils them, or the zombie iterates to their next victim's address.

Greylisting mechanisms maintain a rolling record of connecting SMTP hosts,
anticipating a retry to deliver legitimate mail, within a prescribed time slot.

With the advent of round robin outbound SMTP mail clients clashing with the
extremely effective anti-spam greylisting technique (due to the sender's message
being retried from a cluster of servers, at random, at random times) greylisting
may prevent delivery as these hosts may not retry from the same machine, within
the configured grey to white list exemption time period.

Before the development of ungrey-robins, the author refused to whitelist large
extremely dubious SPF network blocks published by popular senders. He would not
trust large corporation's entire networks of personnel and sales department PCs,
routers, web servers, tape silos, visitor lobby WiFi zones as valid senders. In
February 2015 Paul de Weerd calculated Google published 217,088 IPv4 addresses,
and 29,710,560,942,849,126,597,578,981,376 IPv6 addresses as valid SMTP servers.
See: http://marc.info/?l=openbsd-misc&m=142478407909186
It would be resonable to state Google (and others) have populated SPF with LIES.

Rather, the author found it workable to simply bump spamd's greylist expire time
from 4 hours to 4 days.  Round robin SMTP clients would eventually pass grey
listing, without resorting to manual maintenance of whitelists. Some users were
unwilling to understand delay, not realising email is not an instant messaging
protocol, but a retrying robust one, designed to survive temporary outages.


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Solution
# ========
#

Running ungrey-robins at the same frequency as spamd's greylisting expire time
(4 hours by default) is entirely adequate. There is no benefit in consuming CPU,
I/O & network resources in running it more often, nor is there benefit in
refactoring the script into a daemon. Scheduling intermittent runs of the
lightweight korn shell script via cron has proved to be effective simplicity.

Various sample syslog extracts can be seen in the supplied logs directory.

The decision to whitelist a configurable defined subset of greylisted senders is
achieved by initially insisting the client has a DNS PTR record, and either the
IP's DNS PTR record matches the HELO host name, or the IP's DNS PTR record
matches the IP address. Then the decision to upgrade from grey to whitelisted
status is based on an accumulated score of multiple DNS whitelist lookups.

These clients are spamd whitelisted by ungrey-robins, for 36 days by default,
(which can be considerably extended), so connections from these pooled clients
avoid spamd's greylisting during that period, to infinity and beyond.

Failure to pass any of these tests simply leaves the host to flap about in the
conventional greylisted state, which it may randomly pass in the conventional
manner before the expire period exhausts. No blacklisting is done. Zero damage.


#
# =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
# Source code access
# ==================
#

#
# Home page & HTTP download:
#
http://web.Britvault.Co.UK/products/ungrey-robins/

#
# Anonymous CVS checkout:
# *** NOTE: Neither password nor network encryption is needed.
#
you$ cd $(mktemp -d)
you$ export CVSROOT=':pserver:anoncvs@src.Britvault.Co.UK:/var/spool/cvs'
you$ cvs -q checkout -P ungrey-robins

#
# When done editing, generate a diff of your work, then
# send it with your comments to the mailing list below:
#

you$ cvs -q diff -uNp ungrey-robins/ | tee ungrey-robins.$$.diff

#
# Mailing list for discussion & diffs/patches:
#

post:	ungrey-robins@Britvault.Co.UK
join:	ungrey-robins+subscribe@Britvault.Co.UK
leave:	ungrey-robins+unsubscribe@Britvault.Co.UK