Showing posts with label ssl. Show all posts
Showing posts with label ssl. Show all posts

2014-02-21

Securing Kibana and Elasticsearch with HTTPS/SSL




Encrypting Kibana and Elasticsearch web connections with SSL.

In a previous blog, I covered setting up Elasticsearch, Logstash and Kibana with Lumberjack. Kibana is the web-based front-end to Elasticsearch. By default it will use a standard, unencrypted HTTP connection to communicate with Elasticsearch. This post covers ensuring that all communication to and from Kibana and Elasticsearch are encrypted using SSL.

>>>Click here to do all this with just one Copy & Paste<<<

Enabling SSL for Kibana

This will force an HTTPS connection from the browser to the web server.

SSL Apache support was already turned on by default in our EC2 instances. Check for /etc/httpd/conf.d/ssl.conf which will have the settings enabled, or else make sure that there is an httpd config file with these options enabled:

LoadModule ssl_module modules/mod_ssl.so

... and a virtualhost with these options enabled:

SSLEngine on

If these files are missing, check the Addendum at the bottom of this post.

To enable and force SSL on the Kibana site, add this to a dedicated config file, such as /etc/httpd/conf.d/kibana.conf:

<Directory /var/www/html/kibana>
 SSLRequireSSL
</Directory>

Restarting the web server will result in requests to http://yourip/kibana being forced onto https. However, as Kibana will likely be making AJAX requests to http://yourip:9200/, many browsers will block this unsecured connection from a secure web page. It may also be blocked by XSS rules, as it is on a different port.


Enabling SSL for Elasticsearch

This will force the browser to use SSL to talk to Elasticsearch.

This method places Apache in between the browser and Elasticsearch as a reverse proxy. The web browser sends ES requests to the Apache server which, in turn, redirects these requests to Elasticsearch. The connection between the browser and Apache can be encrypted over HTTPS as though it were a normal web request. Provided that Apache and Elasticsearch are on the same machine, it means that the unencrypted connection to Elasticsearch cannot be intercepted over the internet.

The reverse proxy accepts the incoming Elasticsearch requests on port 443 (https) and pushes them to Elasticsearch on port 9200, which is what Elasticsearch is expecting. Now that ES requests are on the same port as the Kibana web site, the AJAX requests should go through.

We are going to direct requests to https://yourip/elasticsearch to http://localhost:9200/. Add these to the /etc/httpd/conf.d/kibana.conf:

ProxyRequests off
ProxyPass /elasticsearch/ http://elasticsearchhost:9200/

<Location /elasticsearch/>
 ProxyPassReverse /
 SSLRequireSSL
</Location>

Kibana needs to be told which URL to use to communicate with Elasticsearch. Edit /var/www/html/kibana/config.js

and change

elasticsearch: "http://elasticsearchhost:9200",

to

elasticsearch: "https://kibanahost/elasticsearch",

elasticsearchhost and kibanahost should be replaced with the IP addresses or DNS hostnames of the elasticsearch server and the kibana server. If they are both on the same machine, they can be replaced with localhost

Restart Apache with:

sudo service httpd restart

That should all be working. Make sure that port 443 is allowed through your firewalls.


Addendum

If you could not find some of the files mentioned, it is possible to put all of the config in one file. Create a file such as /etc/httpd/conf.d/ssl.conf and add everything to it:

Listen 443
LoadModule ssl_module modules/mod_ssl.so

<VirtualHost _default_:443>
        DocumentRoot "/var/www/html/kibana"
        ErrorLog logs/ssl_error_log
        TransferLog logs/ssl_access_log
        LogLevel warn

        SSLEngine on
        SSLProtocol all -SSLv2
        SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
        SSLCertificateFile /etc/pki/tls/certs/localhost.crt
        SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

        ProxyRequests off
        ProxyPass /elasticsearch/ http://elasticsearchhost:9200/

        <Location /elasticsearch/>
                ProxyPassReverse /
                SSLRequireSSL
        </Location>

        <Directory /var/www/html/kibana>
                SSLRequireSSL
        </Directory>
</VirtualHost>

Don't forget to change /var/www/html/kibana/config.js so that the elasticsearch: value is https://kibanahost/elasticsearch, replacing kibanahost with the IP address or hostname of the Kibana machine.

You might also need to install mod_ssl:

sudo yum install openssl mod_ssl

I did this when booting from a blank Amazon Linux EC2 AMI. This assumes that there is nothing else being served by SSL on port 443.

2014-02-20

Aggregating, searching and centralising logs with Elasticsearch, Logstash, Kibana and Lumberjack




Setting up an ELK stack with lumberjack (logstash-forwarder) for log processing on EC2/CentOS.

The logs at the company for which I work have grown to such a point where the normal unix tools such as grep, cat and tail are grossly inefficient. After researching various log transports and indexing tools, we decided to go for Elasticsearch, Logstash and Kibana (ELK).

The three tools of ELK are maintained by the same group of people and work well together as a result.

I used the helpful blog post Centralizing Logs with Lumberjack, Logstash, and Elasticsearch by Brian Altenhofel to get up and running, but that post is out of date. Here's how I got a stack up and running on an Amazon EC2 instance running CentOS.


Java

Both Elasticsearch and Logstash use the JVM, so you'll need to install Java if it isn't already installed on your system.

Use yum search to find a package similarly named to java-1.7.0-openjdk.x86_64 : OpenJDK Runtime Environment to choose a version of Java to install. At the time this was written, Java 7 was the latest version; java-1.7.0-openjdk.

sudo yum search java
sudo yum install java-1.7.0-openjdk

Elasticsearch

ES is a highly available, scalable RESTful search service. Built on top of Apache Lucene, it has support for clusters and full text search and works with JSON documents.

Get the latest RPM from the ELK downloads page. Get the DEB for Debian and Debian derivatives such as Ubuntu. One could download the gzipped tarball and setup an init.d config, but I'm not going into that, here.

cd /tmp
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.noarch.rpm
sha1sum elasticsearch-1.2.1.noarch.rpm
sudo yum install elasticsearch-1.2.1.noarch.rpm

Compare the output of sha1sum to the SHA1 hash on the download page and make sure that they match before installing Elasticsearch.

Debian users use dpkg -i elasticsearch-1.2.1.deb, instead of yum.

Altenhofel warns about the volume of requests which your ES server is likely to receive. This translates into big bills for services where you are charged per I/O op.

There's a handful of settings one may want to change in /etc/elasticsearch/elasticsearch.yml, such as listing your cluster machines if you can't use multicast and making sure that the data directory is correct.

The service may already be running. If not, start it.

sudo service elasticsearch start

Logstash

Logstash is used for collecting, parsing, manipulating and storing logs. We use it to turn our text logs into JSON for Elasticsearch and better indexing. We also use it to add, remove and manipulate the log contents.

We use Lumberjack for log transport (instead of logstash, which can also be a transport) for its reduced footprint and its ability to encrypt the logs for transport. To this end, one must generate an SSL certificate on the logstash server.

sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/logstash.key -out /etc/ssl/logstash.pub -nodes -days 365

rsa:4096 is the length of the key in bits and -days 365 is the number of days for which the certificate is valid (365 in this instance). Omit -nodes if you want to protect the private key with a passphrase. The public key, /etc/ssl/logstash.pub is the key which will be put on all of our lumberjack machines.

You'll be asked for information to be put into the certificate. This can be automated with configuration files for when you turn this into a headless install procedure.

Get the latest RPM from the ELK downloads page. Get the DEB for Debian and Debian derivatives such as Ubuntu. One could download the flat JAR and setup an init.d config, as Altenhofel describes in his blog. The commands are different in RHEL/CentOS, eg. Debian's start-stop-daemon is similar to Red Hat's daemon.

wget https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.4.1-1_bd507eb.noarch.rpm
sha1sum logstash-1.4.1-1_bd507eb.noarch.rpm
sudo yum install logstash-1.4.1-1_bd507eb.noarch.rpm

If you require some community-contributed filters such as the grep filter, you will also need the logstash contrib download in addition to the core download.

wget http://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-contrib-1.4.1-1_6e42745.noarch.rpm
sha1sum logstash-contrib-1.4.1-1_6e42745.noarch.rpm
sudo yum install logstash-contrib-1.4.1-1_6e42745.noarch.rpm

Compare the output of sha1sum for both downloads to the SHA1 hashes on the download page and make sure that they match before installing Logstash.

Create and edit /etc/logstash/conf.d/logstash.conf to setup the logstash server. Here is an example configuration:

input {
        lumberjack {
                port => 5000
                ssl_certificate => "/etc/ssl/logstash.pub"
                ssl_key => "/etc/ssl/logstash.key"
                ssl_key_passphrase => "YourSSLKeyPassphrase,IfYouMadeOne"
        }
}

filter {
        if [type] == "apache-access-log" {
                grok {
                        match => { "message" => "%{COMBINEDAPACHELOG}" }
                }
                date {
                        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
                }
                geoip {
                        source => "clientip"
                }
                useragent {
                        source => "agent"
                }
        }
        if [type] == "apache-error-log" {
                drop { }
        }
        if [type] == "wowza-access-log" {
                grok {
                        patterns_dir => "/etc/logstash/patterns"
                        match => [ "message", "%{WOWZAACCESSLOG}" ]
                        add_field => [ "datetime", "%{date} %{time} %{tz}" ]
                }
                date {
                        match => [ "datetime", "yyyy-MM-dd HH:mm:ss Z" ]
                }
        }
}

output {
        elasticsearch_http {
                host => "localhost"
        }
}

I've created my own grok pattern for turning the text input of our Wowza access logs into JSON. There's info on how the grok filter works on the Logstash website. Alternatively, see my post on organising wowza logs with logstash and grok for this pattern.

In this example, apache-error-log types are dropped and not sent to the output.

One can start logstash, now. Keep in mind that it may take a few minutes for Logstash to start up. That's just how it is. To start logstash, one must edit /etc/sysconfig/logstash and change START=false to START=true and then start the service.

sudo service logstash start

Lumberjack — logstash-forwarder

Lumberjack has been renamed to logstash-forwarder.

Logstash-forwarder does not use the JVM. This is one of the improvements of logstash-forwarder to reduce its footprint as compared to Logstash. It also has its own transport mechanism to provide security, low latency and reliability.

Go

The installer for logstash-forwarder requires the binaries for the Go Programming Language be available.

Download the linux Go tarball from the Go download page and untar it.

wget http://go.googlecode.com/files/go1.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.2.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Logstash-forwarder

Logstash-forwarder is installed from its Git repository. You'll need to have git installed.

sudo yum install git
cd /usr/local/
sudo git clone https://github.com/elasticsearch/logstash-forwarder.git
cd logstash-forwarder/
sudo /usr/local/go/bin/go build

Starting the service as a daemon requires adding a file to /etc/init.d/.

#! /bin/sh
#!/bin/bash
#
# sshd          Start up the OpenSSH server daemon
#
# chkconfig: 2345 55 25
# description: Lumberjack (logstash-forwarder) ships system logs off to logstash with encryption.
#
# processname: lumberjack
# pidfile: /var/run/lumberjack.pid

### BEGIN INIT INFO
# Provides: lumberjack
# Required-Start: $network $named $local_fs
# Required-Stop: $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start logstash-forwarder
# Description:       Lumberjack (logstash-forwarder) ships system logs off to logstash with encryption
### END INIT INFO

# source function library
. /etc/rc.d/init.d/functions

PROG_DIR='/usr/local/logstash-forwarder'
PROG="$PROG_DIR/logstash-forwarder"
NAME='lumberjack'
CONFIG="/etc/default/$NAME.json"
LOCKFILE="/var/lock/subsys/$NAME"
PIDFILE="/var/run/$NAME.pid"
OUTPUT_LOGFILE="/var/log/$NAME/output.log"

if [ ! -x $PROG ]
then
 echo "$NAME: $PROG does not exist. " && failure
 exit 5
fi

start() {
 status_quiet
 STATUS=$?
 if [ $STATUS -eq 0 ]
 then
  PID=$(cat "$PIDFILE")
  echo -n "$NAME is already running ($PID). " && failure
  echo
  return 1
 fi
 if [ ! -f $CONFIG ]
 then
  echo -n "Config file $CONFIG does not exist. " && failure
  exit 6
 fi
 echo -n "Starting $NAME: "
 OUTPUT_DIR=$(dirname $OUTPUT_LOGFILE)
 [ -d "$OUTPUT_DIR" ] || mkdir "$OUTPUT_DIR"
 nohup "$PROG" -config="$CONFIG" >"$OUTPUT_LOGFILE" 2>&1 &
 RETVAL=$?
 PID=$!
 if [ $RETVAL -eq 0 ]
 then
  COUNTER=1
  while :
  do
   sleep 1
   grep -q 'Connected to' "$OUTPUT_LOGFILE" && break
   if grep -q 'Failed unmarshalling json' "$OUTPUT_LOGFILE"
   then
    failure
    echo
    echo 'Bad config file.'
    echo "Check the log file $OUTPUT_LOGFILE"
    kill "$PID"
    return 1
   fi
   if [ $COUNTER -gt 29 ]
   then
    failure
    echo
    echo "Could not connect to logstash server after $COUNTER seconds"
    echo "Check the log file $OUTPUT_LOGFILE"
    kill "$PID"
    return 1
   else
    COUNTER=$((COUNTER + 1))
   fi
  done
  if touch "$LOCKFILE"
  then
   success
  else
   failure
  fi
  echo
  echo "$PID" > "$PIDFILE"
  return 0
 else
  failure
  return 1
 fi
}

stop() {
 status_quiet
 STATUS=$?
 if [ ! $STATUS -eq 0 ]
 then
  echo -n "$NAME is not running. " && warning
  echo
  return 2
 fi
 PID=$(cat "$PIDFILE")
 echo -n "Stopping $NAME ($PID): "
 kill "$PID"
 RETVAL=$?
 if [ $RETVAL -eq 0 ]
 then
  rm -f "$LOCKFILE"
  rm -f "$PIDFILE"
  success
  echo
  return 0
 else
  failure
  echo
  return 1
 fi
}

status() {
 if [ ! -s "$PIDFILE" ]
 then
  echo "$NAME is not running."
  return 1
 fi
 PID=$(cat "$PIDFILE")
 if ps -p "$PID" > /dev/null
 then
  echo "$NAME is running ($PID)."
  return 0
 else
  echo "PID file is present, but $NAME is not running."
  return 2
 fi
}

status_quiet() {
 status >/dev/null 2>&1
 return $?
}

case "$1" in
 start)
  start
  RETVAL=$?
  ;;
 stop)
  stop
  RETVAL=$?
  ;;
 restart)
  stop
  start
  ;;
 status)
  status
  ;;
 *)
  echo "Usage: $0 {start|stop|status|restart}"
  RETVAL=2
esac
exit $RETVAL

Saving this to /etc/init.d/lumberjack will give you a service named lumberjack.

There is a configuration file which needs to be created at /etc/default/lumberjack.json. It could be anywhere, but this is where it was specified in the init.d file.

{
        "network": {
                "servers": [
                        "127.0.0.1:5000"
                ],
                "ssl ca": "/etc/ssl/logstash.pub",
                "timeout": 10
        },
        "files": [
                {
                        "paths": [
                                "/var/log/httpd/access_log*"
                        ],
                        "fields": {
                                "type": "apache-access-log"
                        }
                }
        ]
}

Change 127.0.0.1 to the IP address of the logstash server.

The type field allows you to differentiate the sources in logstash and is used for the conditional statements in the logstash config file.

The service file(s) in init.d must be executable in order to be usable:

sudo chmod u+x /etc/init.d/lumberjack

Start the service with:

sudo service lumberjack start

Kibana

Kibana is a graphical front-end for Elasticsearch. It can show you your data in various formats.

Kibana is, as are the other ELK tools, available from the ELK downloads page. It is available as a tarball or ZIP file.

cd /tmp/
wget wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
tar -xzf kibana-3.1.0.tar.gz

Installation is as simple as changing /tmp/kibana-3.1.0/config.js and copying the whole directory to your webserver's html folder.

/** @scratch /configuration/config.js/1
 * == Configuration
 * config.js is where you will find the core Kibana configuration. This file contains parameter that
 * must be set before kibana is run for the first time.
 */
define(['settings'],
function (Settings) {
 

  /** @scratch /configuration/config.js/2
   * === Parameters
   */
  return new Settings({

    /** @scratch /configuration/config.js/5
     * ==== elasticsearch
     *
     * The URL to your elasticsearch server. You almost certainly don't
     * want +http://localhost:9200+ here. Even if Kibana and Elasticsearch are on
     * the same host. By default this will attempt to reach ES at the same host you have
     * kibana installed on. You probably want to set it to the FQDN of your
     * elasticsearch host
     */
    elasticsearch: "http://127.0.0.1:9200",

    /** @scratch /configuration/config.js/5
     * ==== default_route
     *
     * This is the default landing page when you don't specify a dashboard to load. You can specify
     * files, scripts or saved dashboards here. For example, if you had saved a dashboard called
     * `WebLogs' to elasticsearch you might use:
     *
     * +default_route: '/dashboard/elasticsearch/WebLogs',+
     */
    default_route     : '/dashboard/file/default.json',

    /** @scratch /configuration/config.js/5
     * ==== kibana-int
     *
     * The default ES index to use for storing Kibana specific object
     * such as stored dashboards
     */
    kibana_index: "kibana-int",

    /** @scratch /configuration/config.js/5
     * ==== panel_name
     *
     * An array of panel modules available. Panels will only be loaded when they are defined in the
     * dashboard, but this list is used in the "add panel" interface.
     */
    panel_names: [
      'histogram',
      'map',
      'pie',
      'table',
      'filtering',
      'timepicker',
      'text',
      'hits',
      'column',
      'trends',
      'bettermap',
      'query',
      'terms',
      'stats',
      'sparklines'
    ]
  });
});

Change the elasticsearch IP address to the IP address of the elasticsearch server, not localhost or 127.0.0.1.

Now copy the whole folder into your web root.

sudo cp -R kibana-3.1.0/ /var/www/html/
sudo mv /var/www/html/kibana-3.1.0/ /var/www/html/kibana
sudo chown -R apache:apache /var/www/html/kibana

Don't forget to change the permissions of this kibana folder so that Apache (or others) can read it.


Firewall

In order to access Kibana and to facilitate log transfer, one must open some ports in the firewall.

PortProtocolReason
9300 – 9400TCPElasticsearch node communication. Other ES nodes used this port to send and receive data. It will try to use 9300 unless it is busy.
9200 – 9300TCPElasticsearch HTTP traffic. Kibana uses this port to communicate with ES. ES will try to use 9200 unless it is busy.
5000TCPWe configured logstash to listen on this port for logs from lumberjack. Add any ports which you need.
80TCPHTTP to access Kibana

2013-07-07

Setting up a mail server with Ubuntu 12.04, Postfix and Courier

I have a domain. I wanted to send and receive e-mail on said domain. All I want is a small, simple mail server for one e-mail address with the option to add more later.

There are a plethora of guides on the web covering the setting up of mail servers. I tried a few and ended up rebooting back to the stock image to start again. Eventually, I got it set up. Here's how I did it.

In the Beginning

I have a VPS from MiniVPS which doesn't have much in the way of resources. I chose to have it pre-loaded with Ubuntu 12.04 LTS (minimal) in the hope that:

  1. There'd be plenty of updated software.
  2. There'd be plenty of support.
  3. There'd be less bloatware in the minimal version.

Add a user

The first thing I do when I get a new server is to add a new user so that I don't have to use root. This can be achieved with:

root@myserver:/# adduser <username>
Adding user `<username>' ...
Adding new group `<username>' (1001) ...
Adding new user `<username>' (1001) with group `<username>' ...
Creating home directory `/home/<username>' ...
Copying files from `/etch/skel/' ...
Enter new UNIX password: <password>
Retype new UNIX password: <password>
Changing the user infromation for <username>
Enter the new value, or press ENTER for the default
        Full Name []: < These details are optional>
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]

Then add that user to the sudo group to allow use of the 'sudo' command:

root@myserver:/# adduser <username> sudo
Adding user `<username>' to group `sudo' ...
Adding user <username> to group sudo
Done.

To allow the user to use the sudo command, the sudo binary needs its permissions modified:

root@myserver:/# chmod u+s /usr/bin/sudo

Now the new user can user sudo to get superuser privileges.

Firewall

The very next thing I like to do is to lock down the firewall. I like to block everything by default and choose what to allow.

I have a reasonable idea of what I am doing with iptables, so I edit a file directly. This is the preamble to my rules:

root@myserver:/# vim /etc/iptables.rules
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:BLACKLIST - [0:0]
:LOGGING - [0:0]

Note: press i to start editing in vim.

Firstly, allow all connections on the local, loop-back interface. This is for internal goings on.

# Allow loopback
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

Secondly, drop all malformed packets. eth0 is the name of the interface receiving connections from the internet, so replace it with yours if need be. In my rules, my interface is called venet0.

I like my rules to be specific, so I specify the interface. It is not necessary to do this.

# Block malformed packets
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags ACK,URG URG -j DROP

The next rule is very important. Without it it would be impossible to connect to the remote server over SSH.

# Allow incoming SSH
-A INPUT -i eth0 -p tcp --dport ssh -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport ssh -m state --state ESTABLISHED -j ACCEPT

This rule allows new SSH connections and established SSH sessions in whilst only allowing established connections out.

All of my rules have a corresponding rule in the other direction for established connections. Many people simply allow all related and established connections, which will simplify the firewall rules.

I have to use the state match (-m state) because my provider doesn't give me access to the conntrack extension. It seems to work just fine, though.

I also allow outbound domain and http connections so that apt-get can work:

# Allow outbound DNS
-A OUTPUT -o eth0 -p udp --dport domain -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p udp --sport domain -m state --state ESTABLISHED -j ACCEPT
# Allow outbound HTTP
-A OUTPUT -o eth0 -p tcp --dport http -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport http -m state --state ESTABLISHED -j ACCEPT

To finish up, I filter everything else through the BLACKLIST chain and finally the LOGGING chain to log dropped packets:

# Fall through to BLACKLIST
-A INPUT -i eth0 -j BLACKLIST
-A OUTPUT -o eth0 -j BLACKLIST
# Blacklist
-A BLACKLIST -j RETURN
# Log unmatched connections
-A INPUT -i eth0 -j LOGGING
-A OUTPUT -o eth0 -j LOGGING
# Log packets on LOGGING
-A LOGGING -m limit --limit 5/min --limit-burst 5 -j LOG --log-prefix "Dropped: " --log-level 7
-A LOGGING -j RETURN
# End
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

Note: press Esc to stop editing in vim and type :w to save and :q to exit (:wq will save and quit).

That's it for now. The limit match prevents the logs from being flooded with entries and the log-prefix makes it easier to identify the entries in the log.

To follow the log in real time, I use this command:

root@myserver:/# watch 'dmesg | tail -10'

Load these rules into iptables using this command:

root@myserver:/# iptables-restore < /etc/iptables.rules

To load these rules automatically after a system restart:

root@myserver:/# vim /etc/network/if-pre-up.d/iptablesload
#!/bin/bash
iptables-restore < /etc/iptables.rules
ip6tables-restore < /etc/ip6tables.rules
exit 0

I also have a file, /etc/ip6tables.rules, which is just blocking everything until I get around to sorting out IPv6.

SSH

Now to make some changes to SSH to tighten security. Edit this file and find these lines:
root@myserver:/# vim /etc/ssh/sshd_config
Protocol 2
LoginGraceTime 20
PermitRootLogin no
PermitEmptyPasswords no
AllowUsers <username>

The values may be different and the final AllowUsers may not even be present. The settings given here will give the SSH some extra protection. Make sure that <username> exists.

Once this is done, its time to restart SSH, log out of the root user and log in with the new user.

root@myserver:/# service ssh restart
root@myserver:/# exit

Getting ready

After logging in with the new user (in this case he'll be called steven), try these commands to check that all is well and that the firewall rules are OK:

steven@myserver:/home/steven# sudo apt-get update
[sudo] password for steven:
Hit http://archive.canonical.com precise Release.gpg
Hit http://archive.canonical.com precise Release
...
Get:33 http://archive.ubuntu.com precise-updates/universer Translation-en [123 kB]
Fetched 3263 kB in 2s (1281 kB/s)
Reading package lists... Done
steven@myserver:/home/steven# sudo su
root@myserver:/home/steven# apt-get upgrade
... <This upgrade step is optional and may take a long time to finish> ...
root@myserver:/home/steven# dpkg-reconfigure tzdata
... <My server was set to Moscow time, so this command let me reset it to GMT> ...

All set. Stay as this root user. I tried using a whole load of sudo commands and it just didn't work.

Postfix

I followed the Postfix guide in the Community Ubuntu Documentation. I'll also record it here.

Installation

Stop sendmail and install postfix:

root@myserver:/home/steven# service sendmail stop
* Stopping Mail Transport Agent (MTA) sendmail     [ OK ]
root@myserver:/home/steven# apt-get install postfix

Accept the defaults in the setup questions for now.

Configuration

I want my mail server to be mail.example.com, but the server is actually called myserver and is at myserver.example.com. This step is optional:

root@myserver:/home/steven# vim /etc/mailname
mail.example.com

In the configuration example, I have included both myserver.example.com (my server's hostname) and mail.example.com (my server's mailname). Whether you do depend on your circumstances.

To reconfigure postfix:

root@myserver:/home/steven# dpkg-reconfigure postfix
 * Stopping Postfix Mail Transport Agent postfix   [ OK ]
You should select these options, or something similar for your needs (change example.com to your domain):
  • General type of mail configuration: Internet Site
  • System mail name: mail.example.com (this could have been myserver.example.com)
  • Root and postmaster mail recipient: <some admin user> (steven)
  • Other destinations to accept mail for (blank for none): mail.example.com, myserver.example.com, example.com, localhost.example.com, localhost
  • Force Synchronous updates on mail queue: <No>
  • Local networks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
  • Use procmail for delivery? <No>
  • Mailbox size limit (bytes): 0
  • Local address extension character: +
  • Internet protocols to use: all

It will apply all of these settings and you should see something like this:

setting synchronous mail queue update: false
setting myorigin
(...)
Running newaliases
 * Stopping Postfix Mail Transport Agent postfix  [ OK ]
 * Starting Postfix Mail Transport Agent postfix  [ OK ]

Authentication

To configure Postfix to use SASL for authentication:

postconf -e 'smtpd_sasl_local_domain ='
postconf -e 'smtpd_sasl_auth_enable = yes'
postconf -e 'smtpd_sasl_security_options = noanonymous'
postconf -e 'broken_sasl_auth_clients = yes'
postconf -e 'smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination'
postconf -e 'inet_interfaces = all'

Note: If you're wondering where 'root@myserver:/home/steven #' went, I removed it here to allow for copying the whole block. You can paste all of the commands at once and Ubuntu will automatically work with it.

Edit /etc/postfix/sasl/smtpd.conf, which may be empty to start with, so it has:

root@myserver:/home/steven# vim /etc/postfix/sasl/smtpd.conf
pwcheck_method: saslauthd
mech_list: plain login

Generate the certificates for encryption:

root@myserver:/home/steven# touch smtpd.key
root@myserver:/home/steven# chmod 600 smtpd.key
root@myserver:/home/steven# openssl genrsa 4096 > smtpd.key
Generating RSA private key, 4096 bit long modulus
...........................................................++
.........................++
e is 65537 (0x10001)
root@myserver:/home/steven# openssl req -new -key smtpd.key -x509 -days 3650 -out smtpd.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:<country_code> (eg. GB)
State or Province Name (full name) [Some-State]:<state, province, county name> (eg. CEREDIGION)
Locality Name (eg, city) []:<city> (eg. ABERYSTWYTH)
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<company name> (I put my own name here)
Organizational Unit Name (eg, section) []:OU name (I put 'E-mail' here)
Common Name (e.g. server FQDN or YOUR name) []:myserver.example.com (see note)
Email Address []:e-mail address
root@myserver:/home/steven# openssl req -new -x509 -extensions v3_ca -keyout cakey.pem -out cacert.pem -days 3650
Generating a 1024 bit RSA private key
........++++++
......................................++++++
writing new private key to 'cakey.pem'
Enter PEM pass phrase:<make sure you can remember this passphrase>
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]: (you know how this goes)
(...)
Email Address []:

Note: the certificate Common Name (CN) should match the one you will use to connect to the server. I connect to mail.example.com, so I use that for the FQDN for the Common Name.

Move them to the correct folders:

mv smtpd.key /etc/ssl/private/
mv smtpd.crt /etc/ssl/certs/
mv cakey.pem /etc/ssl/private/
mv cacert.pem /etc/ssl/certs/

More configuration for TLS in both directions (change mail.example.com):

postconf -e 'smtp_tls_security_level = may'
postconf -e 'smtpd_tls_security_level = may'
postconf -e 'smtpd_tls_auth_only = no'
postconf -e 'smtp_tls_note_starttls_offer = yes'
postconf -e 'smtpd_tls_key_file = /etc/ssl/private/smtpd.key'
postconf -e 'smtpd_tls_cert_file = /etc/ssl/certs/smtpd.crt'
postconf -e 'smtpd_tls_CAfile = /etc/ssl/certs/cacert.pem'
postconf -e 'smtpd_tls_loglevel = 1'
postconf -e 'smtpd_tls_received_header = yes'
postconf -e 'smtpd_tls_session_cache_timeout = 3600s'
postconf -e 'tls_random_source = dev:/dev/urandom'
postconf -e 'myhostname = mail.example.com' # match your mailname

Again, you can copy and paste this whole block of commands.

Now restart postfix.

root@myserver:/home/steven# service postfix restart
 * Stopping Postfix Mail Transport Agent postfix  [ OK ]
 * Starting Postfix Mail Transport Agent postfix  [ OK ]

Install libsasl2-2, sasl2-bin and libsasl2-modules (it is likely that some of them are already installed):

root@myserver:/home/steven# apt-get install libsasl2-2 sasl2-bin libsasl2-modules

Edit /etc/default/saslauthd and do this:

  • Ensure that START=yes is uncommented (has no # before it on the same line)
  • After NAME="saslauthd", add these 3 lines:
    • PWDIR="/var/spool/postfix/var/run/saslauthd"
    • PARAMS="-m ${PWDIR}"
    • PIDFILE="${PWDIR}/saslauthd.pid"
  • Comment out the OPTIONS="-c -m /var-run/saslauthd" line and add OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Next, run this command

root@myserver:/home/steven# dpkg-statoverride --force --update --add root sasl 755 /var/spool/postfix/var/run/saslauthd

Ignore the error about "/var/spool/postfix/var/run/saslauthd" directory does not exist. It will be fixed when the daemon is restarted:

root@myserver:/home/steven# service saslauthd restart

Testing

To check that it is all working, try this:

root@myserver:/home/steven# telnet localhost 25
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix (Ubuntu)
ehlo localhost
250-mail.example.com
(...)
250-STARTTLS
250-AUTH PLAIN LOGIN
(...)
quit
221 2.0.0 Bye
Connection closed by foreign host.

If you see 250-STARTTLS and 250-AUTH then everything looks good.

Sometimes, telnet just sits there when I try telnet localhost 25, but I find that using 127.0.0.1 instead of localhost works, too.

Amavis — antivirus and spam filtering

I followed the PostfixAmavisNew guide from the Community Ubuntu Documentation, again.

This adds Amavis-new, clamav and spamassassin and gets them working with postfix.

Installation

These are the main packages needed:

root@myserver:/home/steven# apt-get install amavisd-new spamassassin clamav-daemon

These packages provide better spam protection and better scanning of archives:

root@myserver:/home/steven# apt-get install libnet-dns-perl libmail-spf-query-perl pyzor razor
root@myserver:/home/steven# apt-get install arj bzip2 cabextract cpio file gzip lha nomarch pax rar unrar unzip unzoo zip zoo

These packages are optional, but will provide better protection.

Configuration

ClamAV

Add clamav to the amavis group and vice versa.

root@myserver:/home/steven# adduser clamav amavis
Adding user `clamav' to group `amavis' ...
Adding user clamav to group amavis
Done.
root@myserver:/home/steven# adduser amavis clamav
Adding user `amavis' to group `clamav' ...
Adding user amavis to group clamav
Done.

Spamassassin

Edit /etc/default/spamassassin and make sure these two settings are set:

  • ENABLED=1 (near the top)
  • CRON=1 (at the bottom)

Start spamassassin:

root@myserver:/home/steven# service spamassassin restart
Restarting SpamAssassin Mail Filter Daemon: spamd.

Amavis

Edit /etc/amavis/conf.d/15-content_filter_mode so it has this for its whole contents:

use strict;

# You can modify this file to re-enable SPAM checking through spamassassin
# and to re-enable antivirus checking.

#
# Default antivirus checking mode
# Uncomment the two lines below to enable it
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);


#
# Default SPAM checking mode
# Uncomment the two lines below to enable it
#

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

1;  # insure a defined return

Restart amavis:

root@myserver:/home/steven# service amavis restart
Stopping amavisd: amavisd-new.
Starting amavisd: amavisd-new.

Postfix

Add an option to /etc/postfix/main.cf using:

root@myserver:/home/steven# postconf -e 'content_filter = smtp-amavis:[127.0.0.1]:10024'

Add this to the end of /etc/postfix/master.cf:

smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Find the line 'pickup' and add two lines to it so that it looks like this:

pickup    fifo  n       -       -       60      1       pickup
         -o content_filter=
         -o receive_override_options=no_header_body_checks

Reload postfix:

root@myserver:/home/steven# service postfix reload
 * Reloading Postfix configuration...             [ OK ]

That should have done it.

Testing

To check that things are working, let's use telnet, again:

root@myserver:/home/steven# telnet localhost 10024
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
quit
221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel
Connection closed by foreign host.

You can check messages for these headers, too:

  • X-Spam-Level:
  • X-Virus-Scanned: Debian amavisd-new at mail.example.com
  • X-Spam-Status: No, hits=-2.3 tagged_above=-1000.0 required=5.0 tests=AWL, BAYES_00
  • X-Spam-Level:

Courier

This is a basic installation of Courier which will only deliver mail to users with user accounts.

Installation

E-mail can work just fine with this one package. Install this package:

root@myserver:/home/steven# apt-get install courier-imap-ssl

If you want IMAP without SSL, too, then add 'courier-imap' to that command.

If you want POP3 (ugh) then there are 'courier-pop-ssl' and 'courier-pop' packages.

Configuration

Add these directories to the skel directory so that they will be created by default for future users:

maildirmake /etc/skel/Maildir
maildirmake /etc/skel/Maildir/.Drafts
maildirmake /etc/skel/Maildir/.Sent
maildirmake /etc/skel/Maildir/.Trash
maildirmake /etc/skel/Maildir/.Templates

Copy and paste the whole block.

If you intend to use any existing users, they will need to have the directories created (change 'steven'):

root@myserver:/home/steven# cp -r /etc/skel/Maildir /home/steven/
root@myserver:/home/steven# chown -R steven:steven /home/steven/Maildir
root@myserver:/home/steven# chmod -R 700 /home/steven/Maildir

That should give a working system. Of course, you'll need to open up the firewall to connect from the internet.

Appendix

  1. SSH — ports to open
  2. Create your own SSL certificated for IMAPS

SSH

Mail ports

These can either be written as separate rule for fine-grained control, or lumped together.

The former:

# Allow inbound IMAPS
-A INPUT -i eth0 -p tcp --dport imaps -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport imaps -m state --state ESTABLISHED -j ACCEPT
# Allow inbound submission
-A INPUT -i eth0 -p tcp --dport submission -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport submission -m state --state ESTABLISHED -j ACCEPT
# Allow inbound SMTP
-A INPUT -i eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT
# Allow outbound SMTP
-A OUTPUT -o eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT

The latter:

# Allow inbound IMAPS, submission and SMTP
-A INPUT -i eth0 -p tcp -m multiport --dports imaps,submission,smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m multiport --sports imaps,submission,smtp -m state --state ESTABLISHED -j ACCEPT
# Allow outbound SMTP
-A OUTPUT -o eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT

Note: don't add both of these sets of rules. Only one is needed.

You can, of course, add other ports such as imap, pop3s and pop3.

IMAP-SSL certificates — creating your own

To create your own SSL certificates in place of the default courier ones:

root@myserver:/home/steven# openssl req -x509 -newkey rsa:4096 -keyout "/etc/ssl/private/courier-imapd-ssl.pem" -out "/etc/ssl/private/courier-imapd-ssl.pem" -nodes -days 1825
root@myserver:/home/steven# openssl req -new -outform PEM -out "/etc/ssl/private/courier-imapd-ssl.crt" -newkey rsa:4096 -nodes -keyout "/etc/ssl/preivate/courier-imapd-ssl.key" -keyform PEM -days 1825 -x509

Edit /etc/courier/imapd-ssl and change TLS_CERTFILE to /etc/ssl/private/courier-imapd-ssl.pem

Some housekeeping and restart the service:

root@myserver:/home/steven# chmod 640 /etc/ssl/private/courier-imapd-ssl.*
root@myserver:/home/steven# chgrp ssl-cert /etc/ssl/private/courier-imapd-ssl.*
root@myserver:/home/steven# service courier-imap-ssl restart
 * Stopping Courier IMAP-SSL server imapd-ssl     [ OK ]
 * Starting Courier IMAP-SSL server imapd-ssl     [ OK ]