Showing posts with label howto. Show all posts
Showing posts with label howto. Show all posts

2014-06-17

Add Configuration Test to Logstash Running As A Service


When running Logstash as a service, there is no option to run the --configtest flag. Here's how to add it to the logstash service.

When making changes to the logstash config, if you were to start the Logstash service using service logstash start, the service will start up (it takes a while to do so) and then crash silently if there are errors in the configuration files. It is possible to check the config files beforehand and get a verbose output by using the --configtest flag when starting logstash.

Service Method

Find the logstash service file at /etc/init.d/logstash and add the function configtest after the force_stop function and change the case switch:

...
force_stop() {
  if status ; then
    stop
    status && kill -KILL `cat "$pidfile"`
  fi
}

configtest() {
  args="$args --configtest"
  nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP / sh -c "
    cd $LS_HOME
    exec \"$program\" $args
  "
}

case "$1" in
  start)
    status
    code=$?
    if [ $code -eq 0 ]; then
      echo "$name is already running"
    else
      start
      code=$?
    fi
    exit $code
    ;;
  stop) stop ;;
  force-stop) force_stop ;;
  status)
    status
    code=$?
    if [ $code -eq 0 ] ; then
      echo "$name is running"
    else
      echo "$name is not running"
    fi
    exit $code
    ;;
  restart)

    stop && start
    ;;
  configtest)
    configtest
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|restart|configtest}" >&2
    exit 3
  ;;
esac

exit $?

This will allow you to run a configuration test without starting the logstash program. Use service logstash configtest to run a config test. It takes a while, so be patient. You might need to prefix it with sudo.

Alternative Method — Direct Run

For completeness, it is possible to run the logstash binary directly to run a config test. The method above simplifies the process.

Use this command to run it directly (you may need to prefix it with sudo):

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest

2014-06-11

Organising Wowza Logs with Logstash and Grok


Pre-processing Wowza Streaming Engine (or Media Server) logs with Grok patterns syntax for better parsing with logstash.

>>>Click here to jump to the pattern to Copy & Paste.<<<

In a previous blog post I described setting up Logstash with Elasticsearch, Kibana and logstash-forwarder (lumberjack). In my logstash config file, I referenced a grok pattern for parsing a Wowza access log. I will describe that pattern, here.

This was created for Wowza Media Server 3.6 and works with Wowza Streaming Engine 4.0.

Grok

Using the grok filter in Logstash allows for patterns to be captured from a logstash field. These captured patterns can then become fully-fledged fields. Instead of a field called message containing the whole Wowza log entry, it can be expanded so that each part of the message becomes a field.

A Wowza log entry looks like this:

2014-06-11\t06:53:34\tUTC\tcreate\tstream\tINFO\t200\t-\t-\t_defaultVHost_\tvod\t_definst_\t0.0\tstream.example.com\t1935\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4?session_token=xxxxx&username=steven\t219.92.86.40\trtsp\t-\tSonyC6603 Build/10.5.A.0.230 stagefright/1.2 (Linux;Android 4.4.2)\t1335194632\t0\t0\t3\t0\t0\t0\tamazons3/my-bucket/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven\t-\t-\t-\t-\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4?session_token=xxxxx&username=steven\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven

It's not very useful like this. I can't easily search through it. I certainly can't see information at a glance. Each of these tab-delimited fields means something. These "columns" have names which are specified in the user manual.

With grok, we can create new fields using the names in the Wowza manual and the values in the message field. Instead of a single message field, we can have:

date
2014-06-11
time
06:53:34

tz



UTC

xEvent



create

xCategory



stream

xSeverity



INFO

xStatus



200

xComment



-

xApp



vod

xAppinst



_definst_

sUri



rtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4?session_token=xxxxx&username=steven

...and so on. This is much better.

Basic Patterns

Creating grok patterns is described in the logstash filters documentation.

Date

The default grok date formats do not include the YYYY-MM-DD format which Wowza uses. We can use the default year, month number and day number.

YMDDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}

This gives a syntax called YMDDATE.


Fields

The fields in the log are separated by tab characters by default. Anything after the date and time AND starts with a tab character is a new field.

WOWZENTRY [^\t]+

This becomes a problem when the xComment field can be written to in your own Wowza Java modules through WMSLogger.info() and similar methods. As that field can contain user data (things not written by Wowza), then it is not unreasonable to expect tab characters in that field.

By that reasoning, I added another, special capture after the xComment capture:

(?<xCommentMalformed>\t.*)?

This prevents tab characters in xComment from causing the grok pattern to break and start the middle of the xComment as other fields. It adds the offending part of the comment to a new field called xCommentMalformed. The field can have any unique name.


Specific fields

There are fields in the log which could be captured by the grok syntax defaults, such as IP, PATH and NONNEGINT. These cannot be used because Wowza will replace the values in these fields with a hyphen (-) if they were not populated by the message.

To address this issue, I created some specific implementations of these syntax labels.

WOWZIP (?:%{IP}|-)
WOWZNONNEGINT (?:%{NONNEGINT}|-)
WOWZPATH (?:%{PATH}|-)
WOWZNUMBER (?:%{NUMBER}|-)


The Full Pattern

The part you probably just wanted to copy and paste.

This creates a syntax named WOWZAACCESSLOG.

This pattern is for Wowza Streaming Engine 4.1.

YMDDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}
WOWZENTRY [^\t]+
WOWZIP (?:%{IP}|-)
WOWZNONNEGINT (?:%{NONNEGINT}|-)
WOWZNUMBER (?:%{NUMBER}|-)
WOWZPATH (?:%{PATH}|-)
WOWZAACCESSLOG %{YMDDATE:date}\t%{TIME:time}\t%{TZ:tz}\t%{WOWZENTRY:xEvent}\t%{WORD:xCategory}\t%{LOGLEVEL:xSeverity}\t%{WOWZNONNEGINT:xStatus}\t%{WOWZENTRY:xCtx}\t%{WOWZENTRY:xComment}(?<xcommentmalformed>\t.*)?\t%{WOWZENTRY:xVhost}\t%{WOWZENTRY:xApp}\t%{WOWZENTRY:xAppinst}\t%{NUMBER:xDuration}\t%{WOWZENTRY:sIp}\t%{WOWZNONNEGINT:sPort}\t%{WOWZENTRY:sUri}\t%{WOWZIP:cIp}\t%{WOWZENTRY:cProto}\t%{WOWZENTRY:cReferrer}\t%{WOWZENTRY:cUserAgent}\t%{WOWZENTRY:cClientId}\t%{WOWZNONNEGINT:csBytes}\t%{WOWZNONNEGINT:scBytes}\t%{WOWZENTRY:xStreamId}\t%{WOWZNONNEGINT:xSpos}\t%{WOWZNONNEGINT:csStreamBytes}\t%{WOWZNONNEGINT:scStreamBytes}\t%{WOWZENTRY:xSname}\t%{WOWZENTRY:xSnameQuery}\t%{WOWZPATH:xFileName}\t%{WOWZENTRY:xFileExt}\t%{WOWZNONNEGINT:xFileSize}\t%{WOWZNUMBER:xFileLength}\t%{WOWZENTRY:xSuri}\t%{WOWZENTRY:xSuriStem}\t%{WOWZENTRY:xSuriQuery}\t%{WOWZENTRY:csUriStem}\t%{WOWZENTRY:csUriQuery}

This pattern is for older versions of Wowza Streaming Engine and Wowza Media Server. If a pattern gives you _grokparsefailure tags or other errors, try the other.

YMDDATE %{YEAR}-%{MONTHNUM}-%{MONTHDAY}
WOWZENTRY [^\t]+
WOWZIP (?:%{IP}|-)
WOWZNONNEGINT (?:%{NONNEGINT}|-)
WOWZPATH (?:%{PATH}|-)
WOWZAACCESSLOG %{YMDDATE:date}\t%{TIME:time}\t%{TZ:tz}\t%{WOWZENTRY:xEvent}\t%{WORD:xCategory}\t%{LOGLEVEL:xSeverity}\t%{WOWZNONNEGINT:xStatus}\t%{WOWZENTRY:xCtx}\t%{WOWZENTRY:xComment}(?<xCommentMalformed>\t.*)?\t%{WOWZENTRY:xVhost}\t%{WOWZENTRY:xApp}\t%{WOWZENTRY:xAppinst}\t%{NUMBER:xDuration}\t%{WOWZENTRY:sIp}\t%{WOWZNONNEGINT:sPort}\t%{WOWZENTRY:sUri}\t%{WOWZIP:cIp}\t%{WOWZENTRY:cProto}\t%{WOWZENTRY:cReferrer}\t%{WOWZENTRY:cUserAgent}\t%{WOWZENTRY:cClientId}\t%{WOWZNONNEGINT:csBytes}\t%{WOWZNONNEGINT:scBytes}\t%{WOWZENTRY:xStreamId}\t%{WOWZNONNEGINT:xSpos}\t%{WOWZNONNEGINT:csStreamBytes}\t%{WOWZNONNEGINT:scStreamBytes}\t%{WOWZENTRY:xSname}\t%{WOWZENTRY:xSnameQuery}\t%{WOWZPATH:xFileName}\t%{WOWZENTRY:xFileExt}\t%{WOWZNONNEGINT:xFileSize}\t%{WOWZNONNEGINT:xFileLength}\t%{WOWZENTRY:xSuri}\t%{WOWZENTRY:xSuriStem}\t%{WOWZENTRY:xSuriQuery}\t%{WOWZENTRY:csUriStem}\t%{WOWZENTRY:csUriQuery}

The difference is that x-file-length can be a decimal in newer logs, whereas my old pattern only looks for positive integers.

It's worth noting that these patterns work if your logging fields are in this order:

  1. date
  2. time
  3. tz
  4. x-event
  5. x-category
  6. x-severity
  7. x-statusx-ctx
  8. x-comment
  9. x-vhost
  10. x-app
  11. x-appinst
  12. x-duration
  13. s-ip
  14. s-port
  15. s-uri
  16. c-ip
  17. c-proto
  18. c-referrer
  19. c-user-agent
  20. c-client-id
  21. cs-bytes
  22. sc-bytes
  23. x-stream-id
  24. x-spos
  25. cs-stream-bytes
  26. sc-stream-bytes
  27. x-sname
  28. x-sname-query
  29. x-file-name
  30. x-file-ext
  31. x-file-size
  32. x-file-length
  33. x-suri
  34. x-suri-stem
  35. x-suri-query
  36. cs-uri-stem
  37. cs-uri-query

Using The Pattern In Logstash

I recommend reading a previous blog post for setting up logstash if you haven't already.

Save these patterns to a file. It can be any file as long as you remember the path and that logstash has permission to read it. Try /etc/logstash/patterns/wowza.

Edit your logstash configuration file /etc/logstash/conf.d/logstash.conf. Add a new grok filter:

filter {
    ... other filters ...
    grok {
        patterns_dir => "/etc/logstash/patterns"
        match => [ "message", "%{WOWZAACCESSLOG}" ]
    }
}

Telling logstash the time

Logstash will use the current time to stamp the log entry. In order to tell logstash to use the time in the log message instead, it needs to be told where to find it.

The required date fields are date, time and tz as defined in the WOWZACCESSLOG grok pattern.

Create a new field containing a timestamp (named datetime) using grok's add_field function and then use the date logstash filter to capture it:

grok {
    patterns_dir => "/etc/logstash/patterns"
    match => [ "message", "%{WOWZAACCESSLOG}" ]
    add_field => [ "datetime", "%{date} %{time} %{tz}" ]
}
date {
    match => [ "datetime", "yyyy-MM-dd HH:mm:ss Z" ]
}

Conclusion

This post has covered the creation of custom grok syntax to parse a Wowza access log entry for use with logstash. It will separate the log parts into individual fields. This should make your Wowza logs considerably more useful and search-able, particularly when used with Elasticsearch.

When used in conjuction with my post covering the setting up of Elasticsearch, Logstash, Kibana and logstash-forwarder, these posts should give enough information to allow you to organise your Wowza logs.

Happy Wowza debugging!

P.S. On the subject of debugging, check out my post for debugging your Wowza modules code by stepping through the Java code in NetBeans (or any JIT debugger) remotely.

2014-05-13

JIT debugging compiled Java on Wowza Streaming Engine with NetBeans



How to debug compiled Java code using NetBeans IDE to step through code and how to set-up Wowza Streaming Engine for just-in-time debugging.

In my daily work, I have a Maven project creating Wowza modules. Wowza's software uses reflection to invoke the methods in custom modules. This means that any exceptions which are thrown in these methods are wrapped in Java's java.lang.reflect.InvocationTargetException. Wowza Streaming Engine prints an error message to its logs, but only the message of this exception. The cause — that which is really the problem — is not reported. Anywhere.

To find out what was going on, I needed to use a debugger to step though the code, line by line, to identify the problem code.

On my development machine using NetBeans, I could not identify any problems. The issues only occurred on the test Wowza Streaming Engine, where the code was in its compiled, JAR form.

Here's how I went about using NetBeans on my development machine to debug the compile code "in production" as it was running.


Setting Up Maven to Compile for Debugging

Skip this step if you aren't using Maven.

In your project's pom.xml file, find or create the element project/build/plugins. Find or create the plugin whose artifactId is maven-compiler-plugin. Add configuration options so that it looks something like this:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>2.3.2</version>
  <configuration>
    <source>1.7</source>
    <target>1.7</target>
    <showDeprecation>true</showDeprecation>
    <compilerArgument>-g:source,lines,vars</compilerArgument>
    <debug>true</debug>
    <debuglevel>lines,vars,source</debuglevel>
    <optimize>false</optimize>
  </configuration>
</plugin>

The lines of note are 9, 10 and 11: compilerArgument, debug and debuglevel. These will tell Maven to compile the code and to include the debug symbols, which our debugger will need.


Compiling For Debugging Without Maven

Skip this step if you are using Maven.

You must tell the Java compiler to include the debug symbols when it compiles your code.

Simply supply the switch -g to the compiler.


Setting up Java for Debugging

Java needs to be run with certain switches to tell it to allow for remote debuggers to attach to it. What these switches are depends on the version of Java which is running. You can find this out by running this command:

java -version

The commands to allow remote debugging in the different versions of Java are:

JDK 1.3 and earlier
-Xnoagent -Djava.compiler=NONE -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6006
JDK 1.4
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6006
Newer JDKs
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6006

These are taken from a StackOverflow answer. I have not tried the switches for the earlier versions. If you are needing those, you should consider updating your JDK.

suspend tells Java to pause execution every time the program runs. You probably don't want this and actually want the program to run as normal until you stop it in the debugger. In that case, the argument is n.

address tells Java to listen on this port for attached debuggers. Remember this port as you'll need it later in NetBeans.

To get Wowza to start with these switches, there is a file containing the start-up switches for Java. The XML file is /usr/local/WowzaStreamingEngine/conf/Tune.xml. Add a VMOption to the /Root/Tune/VMOptions element with the switches which your version of Java requires. It should look something like:

<VMOptions>
  <VMOption>-server</VMOption>
  <VMOption>-Djava.net.preferIPv4Stack=true</VMOption>
  <VMOption>-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=6006</VMOption>
  <!-- <VMOption>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"</VMOption> -->
  <!-- <VMOption>-Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252</VMOption> -->
  <!-- <VMOption>-verbose:gc -Xloggc:"${com.wowza.wms.AppHome}/logs/gc_${com.wowza.wms.StartupDateTime}.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime</VMOption> -->
</VMOptions>

The next time that Wowza is started, you should be able to see that Java is listening to port 6006 (or whichever port you chose).


Debugging in NetBeans

This example is using NetBeans IDE 7.4.

In the IDE, open the project which you wish to debug. That this code is on your development machine and the compiled code is on a different server does not matter.

Click on the Debug (shortcut Alt+D) menu item and then Attach Debugger...

Check the values in the Attach pop-up:

FieldValue
Debugger: Java Debugger (JPDA)
Connectior: SocketAttach (Attaches by socket to other VMs)
Transport: dt_socket
Host 127.0.0.1
Port 6006
Timeout [ms]:

Change the Host and Port to point to the server which you want to debug.

It will look something like this:

NetBeans 7.4 Linux Attach Remote Debugger config

After you click on OK, NetBeans will attempt to connect to the Java process on the remote server. Once it is done, the Attaching Debugger progress bar at the bottom will eventually disappear and the standard debugging windows and buttons will be available. NetBeans should look like you are debugging local code.

You can now add breakpoints in real-time. When the executing Java on the remote server reaches these points in its own code, the execution will stop and you can step through the code and it will even evaluate the variables for you.


Conclusion

This guide should have allowed you to debug your compiled Java code remotely and to set-up a Wowza Streaming Engine for debugging.

It may be possible to use this guide as a base for debugging other Java programs.

When I was debugging the cause of my InvocationTargetException errors, I found that try-catch on Exception was not enough to catch the exceptions. I changed this to Throwable, which allowed me to catch everything. Don't leave your code in production catching Throwables; it is not intended for that. Stick with Exception.

Don't forget to switch off debug symbols for production code.

2014-02-21

Securing Kibana and Elasticsearch with HTTPS/SSL




Encrypting Kibana and Elasticsearch web connections with SSL.

In a previous blog, I covered setting up Elasticsearch, Logstash and Kibana with Lumberjack. Kibana is the web-based front-end to Elasticsearch. By default it will use a standard, unencrypted HTTP connection to communicate with Elasticsearch. This post covers ensuring that all communication to and from Kibana and Elasticsearch are encrypted using SSL.

>>>Click here to do all this with just one Copy & Paste<<<

Enabling SSL for Kibana

This will force an HTTPS connection from the browser to the web server.

SSL Apache support was already turned on by default in our EC2 instances. Check for /etc/httpd/conf.d/ssl.conf which will have the settings enabled, or else make sure that there is an httpd config file with these options enabled:

LoadModule ssl_module modules/mod_ssl.so

... and a virtualhost with these options enabled:

SSLEngine on

If these files are missing, check the Addendum at the bottom of this post.

To enable and force SSL on the Kibana site, add this to a dedicated config file, such as /etc/httpd/conf.d/kibana.conf:

<Directory /var/www/html/kibana>
 SSLRequireSSL
</Directory>

Restarting the web server will result in requests to http://yourip/kibana being forced onto https. However, as Kibana will likely be making AJAX requests to http://yourip:9200/, many browsers will block this unsecured connection from a secure web page. It may also be blocked by XSS rules, as it is on a different port.


Enabling SSL for Elasticsearch

This will force the browser to use SSL to talk to Elasticsearch.

This method places Apache in between the browser and Elasticsearch as a reverse proxy. The web browser sends ES requests to the Apache server which, in turn, redirects these requests to Elasticsearch. The connection between the browser and Apache can be encrypted over HTTPS as though it were a normal web request. Provided that Apache and Elasticsearch are on the same machine, it means that the unencrypted connection to Elasticsearch cannot be intercepted over the internet.

The reverse proxy accepts the incoming Elasticsearch requests on port 443 (https) and pushes them to Elasticsearch on port 9200, which is what Elasticsearch is expecting. Now that ES requests are on the same port as the Kibana web site, the AJAX requests should go through.

We are going to direct requests to https://yourip/elasticsearch to http://localhost:9200/. Add these to the /etc/httpd/conf.d/kibana.conf:

ProxyRequests off
ProxyPass /elasticsearch/ http://elasticsearchhost:9200/

<Location /elasticsearch/>
 ProxyPassReverse /
 SSLRequireSSL
</Location>

Kibana needs to be told which URL to use to communicate with Elasticsearch. Edit /var/www/html/kibana/config.js

and change

elasticsearch: "http://elasticsearchhost:9200",

to

elasticsearch: "https://kibanahost/elasticsearch",

elasticsearchhost and kibanahost should be replaced with the IP addresses or DNS hostnames of the elasticsearch server and the kibana server. If they are both on the same machine, they can be replaced with localhost

Restart Apache with:

sudo service httpd restart

That should all be working. Make sure that port 443 is allowed through your firewalls.


Addendum

If you could not find some of the files mentioned, it is possible to put all of the config in one file. Create a file such as /etc/httpd/conf.d/ssl.conf and add everything to it:

Listen 443
LoadModule ssl_module modules/mod_ssl.so

<VirtualHost _default_:443>
        DocumentRoot "/var/www/html/kibana"
        ErrorLog logs/ssl_error_log
        TransferLog logs/ssl_access_log
        LogLevel warn

        SSLEngine on
        SSLProtocol all -SSLv2
        SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW
        SSLCertificateFile /etc/pki/tls/certs/localhost.crt
        SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

        ProxyRequests off
        ProxyPass /elasticsearch/ http://elasticsearchhost:9200/

        <Location /elasticsearch/>
                ProxyPassReverse /
                SSLRequireSSL
        </Location>

        <Directory /var/www/html/kibana>
                SSLRequireSSL
        </Directory>
</VirtualHost>

Don't forget to change /var/www/html/kibana/config.js so that the elasticsearch: value is https://kibanahost/elasticsearch, replacing kibanahost with the IP address or hostname of the Kibana machine.

You might also need to install mod_ssl:

sudo yum install openssl mod_ssl

I did this when booting from a blank Amazon Linux EC2 AMI. This assumes that there is nothing else being served by SSL on port 443.

2014-02-20

Aggregating, searching and centralising logs with Elasticsearch, Logstash, Kibana and Lumberjack




Setting up an ELK stack with lumberjack (logstash-forwarder) for log processing on EC2/CentOS.

The logs at the company for which I work have grown to such a point where the normal unix tools such as grep, cat and tail are grossly inefficient. After researching various log transports and indexing tools, we decided to go for Elasticsearch, Logstash and Kibana (ELK).

The three tools of ELK are maintained by the same group of people and work well together as a result.

I used the helpful blog post Centralizing Logs with Lumberjack, Logstash, and Elasticsearch by Brian Altenhofel to get up and running, but that post is out of date. Here's how I got a stack up and running on an Amazon EC2 instance running CentOS.


Java

Both Elasticsearch and Logstash use the JVM, so you'll need to install Java if it isn't already installed on your system.

Use yum search to find a package similarly named to java-1.7.0-openjdk.x86_64 : OpenJDK Runtime Environment to choose a version of Java to install. At the time this was written, Java 7 was the latest version; java-1.7.0-openjdk.

sudo yum search java
sudo yum install java-1.7.0-openjdk

Elasticsearch

ES is a highly available, scalable RESTful search service. Built on top of Apache Lucene, it has support for clusters and full text search and works with JSON documents.

Get the latest RPM from the ELK downloads page. Get the DEB for Debian and Debian derivatives such as Ubuntu. One could download the gzipped tarball and setup an init.d config, but I'm not going into that, here.

cd /tmp
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.2.1.noarch.rpm
sha1sum elasticsearch-1.2.1.noarch.rpm
sudo yum install elasticsearch-1.2.1.noarch.rpm

Compare the output of sha1sum to the SHA1 hash on the download page and make sure that they match before installing Elasticsearch.

Debian users use dpkg -i elasticsearch-1.2.1.deb, instead of yum.

Altenhofel warns about the volume of requests which your ES server is likely to receive. This translates into big bills for services where you are charged per I/O op.

There's a handful of settings one may want to change in /etc/elasticsearch/elasticsearch.yml, such as listing your cluster machines if you can't use multicast and making sure that the data directory is correct.

The service may already be running. If not, start it.

sudo service elasticsearch start

Logstash

Logstash is used for collecting, parsing, manipulating and storing logs. We use it to turn our text logs into JSON for Elasticsearch and better indexing. We also use it to add, remove and manipulate the log contents.

We use Lumberjack for log transport (instead of logstash, which can also be a transport) for its reduced footprint and its ability to encrypt the logs for transport. To this end, one must generate an SSL certificate on the logstash server.

sudo openssl req -x509 -newkey rsa:4096 -keyout /etc/ssl/logstash.key -out /etc/ssl/logstash.pub -nodes -days 365

rsa:4096 is the length of the key in bits and -days 365 is the number of days for which the certificate is valid (365 in this instance). Omit -nodes if you want to protect the private key with a passphrase. The public key, /etc/ssl/logstash.pub is the key which will be put on all of our lumberjack machines.

You'll be asked for information to be put into the certificate. This can be automated with configuration files for when you turn this into a headless install procedure.

Get the latest RPM from the ELK downloads page. Get the DEB for Debian and Debian derivatives such as Ubuntu. One could download the flat JAR and setup an init.d config, as Altenhofel describes in his blog. The commands are different in RHEL/CentOS, eg. Debian's start-stop-daemon is similar to Red Hat's daemon.

wget https://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-1.4.1-1_bd507eb.noarch.rpm
sha1sum logstash-1.4.1-1_bd507eb.noarch.rpm
sudo yum install logstash-1.4.1-1_bd507eb.noarch.rpm

If you require some community-contributed filters such as the grep filter, you will also need the logstash contrib download in addition to the core download.

wget http://download.elasticsearch.org/logstash/logstash/packages/centos/logstash-contrib-1.4.1-1_6e42745.noarch.rpm
sha1sum logstash-contrib-1.4.1-1_6e42745.noarch.rpm
sudo yum install logstash-contrib-1.4.1-1_6e42745.noarch.rpm

Compare the output of sha1sum for both downloads to the SHA1 hashes on the download page and make sure that they match before installing Logstash.

Create and edit /etc/logstash/conf.d/logstash.conf to setup the logstash server. Here is an example configuration:

input {
        lumberjack {
                port => 5000
                ssl_certificate => "/etc/ssl/logstash.pub"
                ssl_key => "/etc/ssl/logstash.key"
                ssl_key_passphrase => "YourSSLKeyPassphrase,IfYouMadeOne"
        }
}

filter {
        if [type] == "apache-access-log" {
                grok {
                        match => { "message" => "%{COMBINEDAPACHELOG}" }
                }
                date {
                        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
                }
                geoip {
                        source => "clientip"
                }
                useragent {
                        source => "agent"
                }
        }
        if [type] == "apache-error-log" {
                drop { }
        }
        if [type] == "wowza-access-log" {
                grok {
                        patterns_dir => "/etc/logstash/patterns"
                        match => [ "message", "%{WOWZAACCESSLOG}" ]
                        add_field => [ "datetime", "%{date} %{time} %{tz}" ]
                }
                date {
                        match => [ "datetime", "yyyy-MM-dd HH:mm:ss Z" ]
                }
        }
}

output {
        elasticsearch_http {
                host => "localhost"
        }
}

I've created my own grok pattern for turning the text input of our Wowza access logs into JSON. There's info on how the grok filter works on the Logstash website. Alternatively, see my post on organising wowza logs with logstash and grok for this pattern.

In this example, apache-error-log types are dropped and not sent to the output.

One can start logstash, now. Keep in mind that it may take a few minutes for Logstash to start up. That's just how it is. To start logstash, one must edit /etc/sysconfig/logstash and change START=false to START=true and then start the service.

sudo service logstash start

Lumberjack — logstash-forwarder

Lumberjack has been renamed to logstash-forwarder.

Logstash-forwarder does not use the JVM. This is one of the improvements of logstash-forwarder to reduce its footprint as compared to Logstash. It also has its own transport mechanism to provide security, low latency and reliability.

Go

The installer for logstash-forwarder requires the binaries for the Go Programming Language be available.

Download the linux Go tarball from the Go download page and untar it.

wget http://go.googlecode.com/files/go1.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.2.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

Logstash-forwarder

Logstash-forwarder is installed from its Git repository. You'll need to have git installed.

sudo yum install git
cd /usr/local/
sudo git clone https://github.com/elasticsearch/logstash-forwarder.git
cd logstash-forwarder/
sudo /usr/local/go/bin/go build

Starting the service as a daemon requires adding a file to /etc/init.d/.

#! /bin/sh
#!/bin/bash
#
# sshd          Start up the OpenSSH server daemon
#
# chkconfig: 2345 55 25
# description: Lumberjack (logstash-forwarder) ships system logs off to logstash with encryption.
#
# processname: lumberjack
# pidfile: /var/run/lumberjack.pid

### BEGIN INIT INFO
# Provides: lumberjack
# Required-Start: $network $named $local_fs
# Required-Stop: $network $named
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start logstash-forwarder
# Description:       Lumberjack (logstash-forwarder) ships system logs off to logstash with encryption
### END INIT INFO

# source function library
. /etc/rc.d/init.d/functions

PROG_DIR='/usr/local/logstash-forwarder'
PROG="$PROG_DIR/logstash-forwarder"
NAME='lumberjack'
CONFIG="/etc/default/$NAME.json"
LOCKFILE="/var/lock/subsys/$NAME"
PIDFILE="/var/run/$NAME.pid"
OUTPUT_LOGFILE="/var/log/$NAME/output.log"

if [ ! -x $PROG ]
then
 echo "$NAME: $PROG does not exist. " && failure
 exit 5
fi

start() {
 status_quiet
 STATUS=$?
 if [ $STATUS -eq 0 ]
 then
  PID=$(cat "$PIDFILE")
  echo -n "$NAME is already running ($PID). " && failure
  echo
  return 1
 fi
 if [ ! -f $CONFIG ]
 then
  echo -n "Config file $CONFIG does not exist. " && failure
  exit 6
 fi
 echo -n "Starting $NAME: "
 OUTPUT_DIR=$(dirname $OUTPUT_LOGFILE)
 [ -d "$OUTPUT_DIR" ] || mkdir "$OUTPUT_DIR"
 nohup "$PROG" -config="$CONFIG" >"$OUTPUT_LOGFILE" 2>&1 &
 RETVAL=$?
 PID=$!
 if [ $RETVAL -eq 0 ]
 then
  COUNTER=1
  while :
  do
   sleep 1
   grep -q 'Connected to' "$OUTPUT_LOGFILE" && break
   if grep -q 'Failed unmarshalling json' "$OUTPUT_LOGFILE"
   then
    failure
    echo
    echo 'Bad config file.'
    echo "Check the log file $OUTPUT_LOGFILE"
    kill "$PID"
    return 1
   fi
   if [ $COUNTER -gt 29 ]
   then
    failure
    echo
    echo "Could not connect to logstash server after $COUNTER seconds"
    echo "Check the log file $OUTPUT_LOGFILE"
    kill "$PID"
    return 1
   else
    COUNTER=$((COUNTER + 1))
   fi
  done
  if touch "$LOCKFILE"
  then
   success
  else
   failure
  fi
  echo
  echo "$PID" > "$PIDFILE"
  return 0
 else
  failure
  return 1
 fi
}

stop() {
 status_quiet
 STATUS=$?
 if [ ! $STATUS -eq 0 ]
 then
  echo -n "$NAME is not running. " && warning
  echo
  return 2
 fi
 PID=$(cat "$PIDFILE")
 echo -n "Stopping $NAME ($PID): "
 kill "$PID"
 RETVAL=$?
 if [ $RETVAL -eq 0 ]
 then
  rm -f "$LOCKFILE"
  rm -f "$PIDFILE"
  success
  echo
  return 0
 else
  failure
  echo
  return 1
 fi
}

status() {
 if [ ! -s "$PIDFILE" ]
 then
  echo "$NAME is not running."
  return 1
 fi
 PID=$(cat "$PIDFILE")
 if ps -p "$PID" > /dev/null
 then
  echo "$NAME is running ($PID)."
  return 0
 else
  echo "PID file is present, but $NAME is not running."
  return 2
 fi
}

status_quiet() {
 status >/dev/null 2>&1
 return $?
}

case "$1" in
 start)
  start
  RETVAL=$?
  ;;
 stop)
  stop
  RETVAL=$?
  ;;
 restart)
  stop
  start
  ;;
 status)
  status
  ;;
 *)
  echo "Usage: $0 {start|stop|status|restart}"
  RETVAL=2
esac
exit $RETVAL

Saving this to /etc/init.d/lumberjack will give you a service named lumberjack.

There is a configuration file which needs to be created at /etc/default/lumberjack.json. It could be anywhere, but this is where it was specified in the init.d file.

{
        "network": {
                "servers": [
                        "127.0.0.1:5000"
                ],
                "ssl ca": "/etc/ssl/logstash.pub",
                "timeout": 10
        },
        "files": [
                {
                        "paths": [
                                "/var/log/httpd/access_log*"
                        ],
                        "fields": {
                                "type": "apache-access-log"
                        }
                }
        ]
}

Change 127.0.0.1 to the IP address of the logstash server.

The type field allows you to differentiate the sources in logstash and is used for the conditional statements in the logstash config file.

The service file(s) in init.d must be executable in order to be usable:

sudo chmod u+x /etc/init.d/lumberjack

Start the service with:

sudo service lumberjack start

Kibana

Kibana is a graphical front-end for Elasticsearch. It can show you your data in various formats.

Kibana is, as are the other ELK tools, available from the ELK downloads page. It is available as a tarball or ZIP file.

cd /tmp/
wget wget https://download.elasticsearch.org/kibana/kibana/kibana-3.1.0.tar.gz
tar -xzf kibana-3.1.0.tar.gz

Installation is as simple as changing /tmp/kibana-3.1.0/config.js and copying the whole directory to your webserver's html folder.

/** @scratch /configuration/config.js/1
 * == Configuration
 * config.js is where you will find the core Kibana configuration. This file contains parameter that
 * must be set before kibana is run for the first time.
 */
define(['settings'],
function (Settings) {
 

  /** @scratch /configuration/config.js/2
   * === Parameters
   */
  return new Settings({

    /** @scratch /configuration/config.js/5
     * ==== elasticsearch
     *
     * The URL to your elasticsearch server. You almost certainly don't
     * want +http://localhost:9200+ here. Even if Kibana and Elasticsearch are on
     * the same host. By default this will attempt to reach ES at the same host you have
     * kibana installed on. You probably want to set it to the FQDN of your
     * elasticsearch host
     */
    elasticsearch: "http://127.0.0.1:9200",

    /** @scratch /configuration/config.js/5
     * ==== default_route
     *
     * This is the default landing page when you don't specify a dashboard to load. You can specify
     * files, scripts or saved dashboards here. For example, if you had saved a dashboard called
     * `WebLogs' to elasticsearch you might use:
     *
     * +default_route: '/dashboard/elasticsearch/WebLogs',+
     */
    default_route     : '/dashboard/file/default.json',

    /** @scratch /configuration/config.js/5
     * ==== kibana-int
     *
     * The default ES index to use for storing Kibana specific object
     * such as stored dashboards
     */
    kibana_index: "kibana-int",

    /** @scratch /configuration/config.js/5
     * ==== panel_name
     *
     * An array of panel modules available. Panels will only be loaded when they are defined in the
     * dashboard, but this list is used in the "add panel" interface.
     */
    panel_names: [
      'histogram',
      'map',
      'pie',
      'table',
      'filtering',
      'timepicker',
      'text',
      'hits',
      'column',
      'trends',
      'bettermap',
      'query',
      'terms',
      'stats',
      'sparklines'
    ]
  });
});

Change the elasticsearch IP address to the IP address of the elasticsearch server, not localhost or 127.0.0.1.

Now copy the whole folder into your web root.

sudo cp -R kibana-3.1.0/ /var/www/html/
sudo mv /var/www/html/kibana-3.1.0/ /var/www/html/kibana
sudo chown -R apache:apache /var/www/html/kibana

Don't forget to change the permissions of this kibana folder so that Apache (or others) can read it.


Firewall

In order to access Kibana and to facilitate log transfer, one must open some ports in the firewall.

PortProtocolReason
9300 – 9400TCPElasticsearch node communication. Other ES nodes used this port to send and receive data. It will try to use 9300 unless it is busy.
9200 – 9300TCPElasticsearch HTTP traffic. Kibana uses this port to communicate with ES. ES will try to use 9200 unless it is busy.
5000TCPWe configured logstash to listen on this port for logs from lumberjack. Add any ports which you need.
80TCPHTTP to access Kibana

2013-08-21

Creating a Wowza Module — the basics

These simple steps don't seem to be covered anywhere, so here's how I started out developing a module for the Wowza Media Server on a remote machine using Netbeans.


The Java Code

The starting Java to test file placement and settings:

package com.example.wowza.modules;

import com.wowza.wms.application.IApplicationInstance;
import com.wowza.wms.module.ModuleBase;

public class TestModule extends ModuleBase {

    public final static String MODULE_NAME = "TestModule";

    public void onAppStart(IApplicationInstance appInstance) {
        getLogger().info("" + MODULE_NAME + " module loaded with application instance");
    }

    public void onAppStop(IApplicationInstance appInstance) {
        getLogger().info("Application instance was stopped, so " + MODULE_NAME + " module has stopped.");
    }

}

That should be enough to inform that the module was started and, by extension, is in the right place.


To get this to compile in Netbeans, you'll need to get hold of both wms-server.jar and log4j-1.2.17.jar from the lib/ directory on the server. You should import these files as libraries to your local Netbeans project. You won't need to move these files around after this.


Transfer the created JAR file from the local project's dist folder to the lib directory on the server.


I used psftp.exe from the makers of PuTTY to transfer the JAR archives between the server and my Windows local machine in both directions.


The Application

Create a test application with these commands:

mkdir applications/testapp
mkdir conf/testapp
cp conf/Application.xml conf/testapp/

Edit the conf/testapp/Application.xml to add this module definition to the <Modules> section before the closing </Modules> element:

<Module>
    <Name>TestModule</Name>
    <Description>A test module</Description>
    <Class>com.example.wowza.modules.TestModule</Class>
</Module>

Testing

Starting an application to test that the module works:
VLC media player: Media -> Open Network Stream -> rtsp://[server_ip]/testapp/mp4:sample.mp4


Checking the access log at logs/wowzamediaserver_access.log should reveal an entry similar to:

2013-08-21   01:43:19        BST     comment      server  INFO    200     -       TestModule module loaded with application  -       -       - (etc.)

It may be buried by lots of access logs regarding the MP4 video we just loaded. Our servers have the wrong time; I am not nocturnal.


Stopping the server will also record the message from onAppStop() into the same log file.


The module was successfully loaded, so all of the relevant pieces are in place to start developing a useful module. This was mainly for my own reference, but hopefully it helped you, too.

2013-07-07

Setting up a mail server with Ubuntu 12.04, Postfix and Courier

I have a domain. I wanted to send and receive e-mail on said domain. All I want is a small, simple mail server for one e-mail address with the option to add more later.

There are a plethora of guides on the web covering the setting up of mail servers. I tried a few and ended up rebooting back to the stock image to start again. Eventually, I got it set up. Here's how I did it.

In the Beginning

I have a VPS from MiniVPS which doesn't have much in the way of resources. I chose to have it pre-loaded with Ubuntu 12.04 LTS (minimal) in the hope that:

  1. There'd be plenty of updated software.
  2. There'd be plenty of support.
  3. There'd be less bloatware in the minimal version.

Add a user

The first thing I do when I get a new server is to add a new user so that I don't have to use root. This can be achieved with:

root@myserver:/# adduser <username>
Adding user `<username>' ...
Adding new group `<username>' (1001) ...
Adding new user `<username>' (1001) with group `<username>' ...
Creating home directory `/home/<username>' ...
Copying files from `/etch/skel/' ...
Enter new UNIX password: <password>
Retype new UNIX password: <password>
Changing the user infromation for <username>
Enter the new value, or press ENTER for the default
        Full Name []: < These details are optional>
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n]

Then add that user to the sudo group to allow use of the 'sudo' command:

root@myserver:/# adduser <username> sudo
Adding user `<username>' to group `sudo' ...
Adding user <username> to group sudo
Done.

To allow the user to use the sudo command, the sudo binary needs its permissions modified:

root@myserver:/# chmod u+s /usr/bin/sudo

Now the new user can user sudo to get superuser privileges.

Firewall

The very next thing I like to do is to lock down the firewall. I like to block everything by default and choose what to allow.

I have a reasonable idea of what I am doing with iptables, so I edit a file directly. This is the preamble to my rules:

root@myserver:/# vim /etc/iptables.rules
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:BLACKLIST - [0:0]
:LOGGING - [0:0]

Note: press i to start editing in vim.

Firstly, allow all connections on the local, loop-back interface. This is for internal goings on.

# Allow loopback
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

Secondly, drop all malformed packets. eth0 is the name of the interface receiving connections from the internet, so replace it with yours if need be. In my rules, my interface is called venet0.

I like my rules to be specific, so I specify the interface. It is not necessary to do this.

# Block malformed packets
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,RST FIN,RST -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags FIN,ACK FIN -j DROP
-A INPUT -i eth0 -p tcp -m tcp --tcp-flags ACK,URG URG -j DROP

The next rule is very important. Without it it would be impossible to connect to the remote server over SSH.

# Allow incoming SSH
-A INPUT -i eth0 -p tcp --dport ssh -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport ssh -m state --state ESTABLISHED -j ACCEPT

This rule allows new SSH connections and established SSH sessions in whilst only allowing established connections out.

All of my rules have a corresponding rule in the other direction for established connections. Many people simply allow all related and established connections, which will simplify the firewall rules.

I have to use the state match (-m state) because my provider doesn't give me access to the conntrack extension. It seems to work just fine, though.

I also allow outbound domain and http connections so that apt-get can work:

# Allow outbound DNS
-A OUTPUT -o eth0 -p udp --dport domain -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p udp --sport domain -m state --state ESTABLISHED -j ACCEPT
# Allow outbound HTTP
-A OUTPUT -o eth0 -p tcp --dport http -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport http -m state --state ESTABLISHED -j ACCEPT

To finish up, I filter everything else through the BLACKLIST chain and finally the LOGGING chain to log dropped packets:

# Fall through to BLACKLIST
-A INPUT -i eth0 -j BLACKLIST
-A OUTPUT -o eth0 -j BLACKLIST
# Blacklist
-A BLACKLIST -j RETURN
# Log unmatched connections
-A INPUT -i eth0 -j LOGGING
-A OUTPUT -o eth0 -j LOGGING
# Log packets on LOGGING
-A LOGGING -m limit --limit 5/min --limit-burst 5 -j LOG --log-prefix "Dropped: " --log-level 7
-A LOGGING -j RETURN
# End
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT

Note: press Esc to stop editing in vim and type :w to save and :q to exit (:wq will save and quit).

That's it for now. The limit match prevents the logs from being flooded with entries and the log-prefix makes it easier to identify the entries in the log.

To follow the log in real time, I use this command:

root@myserver:/# watch 'dmesg | tail -10'

Load these rules into iptables using this command:

root@myserver:/# iptables-restore < /etc/iptables.rules

To load these rules automatically after a system restart:

root@myserver:/# vim /etc/network/if-pre-up.d/iptablesload
#!/bin/bash
iptables-restore < /etc/iptables.rules
ip6tables-restore < /etc/ip6tables.rules
exit 0

I also have a file, /etc/ip6tables.rules, which is just blocking everything until I get around to sorting out IPv6.

SSH

Now to make some changes to SSH to tighten security. Edit this file and find these lines:
root@myserver:/# vim /etc/ssh/sshd_config
Protocol 2
LoginGraceTime 20
PermitRootLogin no
PermitEmptyPasswords no
AllowUsers <username>

The values may be different and the final AllowUsers may not even be present. The settings given here will give the SSH some extra protection. Make sure that <username> exists.

Once this is done, its time to restart SSH, log out of the root user and log in with the new user.

root@myserver:/# service ssh restart
root@myserver:/# exit

Getting ready

After logging in with the new user (in this case he'll be called steven), try these commands to check that all is well and that the firewall rules are OK:

steven@myserver:/home/steven# sudo apt-get update
[sudo] password for steven:
Hit http://archive.canonical.com precise Release.gpg
Hit http://archive.canonical.com precise Release
...
Get:33 http://archive.ubuntu.com precise-updates/universer Translation-en [123 kB]
Fetched 3263 kB in 2s (1281 kB/s)
Reading package lists... Done
steven@myserver:/home/steven# sudo su
root@myserver:/home/steven# apt-get upgrade
... <This upgrade step is optional and may take a long time to finish> ...
root@myserver:/home/steven# dpkg-reconfigure tzdata
... <My server was set to Moscow time, so this command let me reset it to GMT> ...

All set. Stay as this root user. I tried using a whole load of sudo commands and it just didn't work.

Postfix

I followed the Postfix guide in the Community Ubuntu Documentation. I'll also record it here.

Installation

Stop sendmail and install postfix:

root@myserver:/home/steven# service sendmail stop
* Stopping Mail Transport Agent (MTA) sendmail     [ OK ]
root@myserver:/home/steven# apt-get install postfix

Accept the defaults in the setup questions for now.

Configuration

I want my mail server to be mail.example.com, but the server is actually called myserver and is at myserver.example.com. This step is optional:

root@myserver:/home/steven# vim /etc/mailname
mail.example.com

In the configuration example, I have included both myserver.example.com (my server's hostname) and mail.example.com (my server's mailname). Whether you do depend on your circumstances.

To reconfigure postfix:

root@myserver:/home/steven# dpkg-reconfigure postfix
 * Stopping Postfix Mail Transport Agent postfix   [ OK ]
You should select these options, or something similar for your needs (change example.com to your domain):
  • General type of mail configuration: Internet Site
  • System mail name: mail.example.com (this could have been myserver.example.com)
  • Root and postmaster mail recipient: <some admin user> (steven)
  • Other destinations to accept mail for (blank for none): mail.example.com, myserver.example.com, example.com, localhost.example.com, localhost
  • Force Synchronous updates on mail queue: <No>
  • Local networks: 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
  • Use procmail for delivery? <No>
  • Mailbox size limit (bytes): 0
  • Local address extension character: +
  • Internet protocols to use: all

It will apply all of these settings and you should see something like this:

setting synchronous mail queue update: false
setting myorigin
(...)
Running newaliases
 * Stopping Postfix Mail Transport Agent postfix  [ OK ]
 * Starting Postfix Mail Transport Agent postfix  [ OK ]

Authentication

To configure Postfix to use SASL for authentication:

postconf -e 'smtpd_sasl_local_domain ='
postconf -e 'smtpd_sasl_auth_enable = yes'
postconf -e 'smtpd_sasl_security_options = noanonymous'
postconf -e 'broken_sasl_auth_clients = yes'
postconf -e 'smtpd_recipient_restrictions = permit_sasl_authenticated,permit_mynetworks,reject_unauth_destination'
postconf -e 'inet_interfaces = all'

Note: If you're wondering where 'root@myserver:/home/steven #' went, I removed it here to allow for copying the whole block. You can paste all of the commands at once and Ubuntu will automatically work with it.

Edit /etc/postfix/sasl/smtpd.conf, which may be empty to start with, so it has:

root@myserver:/home/steven# vim /etc/postfix/sasl/smtpd.conf
pwcheck_method: saslauthd
mech_list: plain login

Generate the certificates for encryption:

root@myserver:/home/steven# touch smtpd.key
root@myserver:/home/steven# chmod 600 smtpd.key
root@myserver:/home/steven# openssl genrsa 4096 > smtpd.key
Generating RSA private key, 4096 bit long modulus
...........................................................++
.........................++
e is 65537 (0x10001)
root@myserver:/home/steven# openssl req -new -key smtpd.key -x509 -days 3650 -out smtpd.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:<country_code> (eg. GB)
State or Province Name (full name) [Some-State]:<state, province, county name> (eg. CEREDIGION)
Locality Name (eg, city) []:<city> (eg. ABERYSTWYTH)
Organization Name (eg, company) [Internet Widgits Pty Ltd]:<company name> (I put my own name here)
Organizational Unit Name (eg, section) []:OU name (I put 'E-mail' here)
Common Name (e.g. server FQDN or YOUR name) []:myserver.example.com (see note)
Email Address []:e-mail address
root@myserver:/home/steven# openssl req -new -x509 -extensions v3_ca -keyout cakey.pem -out cacert.pem -days 3650
Generating a 1024 bit RSA private key
........++++++
......................................++++++
writing new private key to 'cakey.pem'
Enter PEM pass phrase:<make sure you can remember this passphrase>
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]: (you know how this goes)
(...)
Email Address []:

Note: the certificate Common Name (CN) should match the one you will use to connect to the server. I connect to mail.example.com, so I use that for the FQDN for the Common Name.

Move them to the correct folders:

mv smtpd.key /etc/ssl/private/
mv smtpd.crt /etc/ssl/certs/
mv cakey.pem /etc/ssl/private/
mv cacert.pem /etc/ssl/certs/

More configuration for TLS in both directions (change mail.example.com):

postconf -e 'smtp_tls_security_level = may'
postconf -e 'smtpd_tls_security_level = may'
postconf -e 'smtpd_tls_auth_only = no'
postconf -e 'smtp_tls_note_starttls_offer = yes'
postconf -e 'smtpd_tls_key_file = /etc/ssl/private/smtpd.key'
postconf -e 'smtpd_tls_cert_file = /etc/ssl/certs/smtpd.crt'
postconf -e 'smtpd_tls_CAfile = /etc/ssl/certs/cacert.pem'
postconf -e 'smtpd_tls_loglevel = 1'
postconf -e 'smtpd_tls_received_header = yes'
postconf -e 'smtpd_tls_session_cache_timeout = 3600s'
postconf -e 'tls_random_source = dev:/dev/urandom'
postconf -e 'myhostname = mail.example.com' # match your mailname

Again, you can copy and paste this whole block of commands.

Now restart postfix.

root@myserver:/home/steven# service postfix restart
 * Stopping Postfix Mail Transport Agent postfix  [ OK ]
 * Starting Postfix Mail Transport Agent postfix  [ OK ]

Install libsasl2-2, sasl2-bin and libsasl2-modules (it is likely that some of them are already installed):

root@myserver:/home/steven# apt-get install libsasl2-2 sasl2-bin libsasl2-modules

Edit /etc/default/saslauthd and do this:

  • Ensure that START=yes is uncommented (has no # before it on the same line)
  • After NAME="saslauthd", add these 3 lines:
    • PWDIR="/var/spool/postfix/var/run/saslauthd"
    • PARAMS="-m ${PWDIR}"
    • PIDFILE="${PWDIR}/saslauthd.pid"
  • Comment out the OPTIONS="-c -m /var-run/saslauthd" line and add OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"

Next, run this command

root@myserver:/home/steven# dpkg-statoverride --force --update --add root sasl 755 /var/spool/postfix/var/run/saslauthd

Ignore the error about "/var/spool/postfix/var/run/saslauthd" directory does not exist. It will be fixed when the daemon is restarted:

root@myserver:/home/steven# service saslauthd restart

Testing

To check that it is all working, try this:

root@myserver:/home/steven# telnet localhost 25
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix (Ubuntu)
ehlo localhost
250-mail.example.com
(...)
250-STARTTLS
250-AUTH PLAIN LOGIN
(...)
quit
221 2.0.0 Bye
Connection closed by foreign host.

If you see 250-STARTTLS and 250-AUTH then everything looks good.

Sometimes, telnet just sits there when I try telnet localhost 25, but I find that using 127.0.0.1 instead of localhost works, too.

Amavis — antivirus and spam filtering

I followed the PostfixAmavisNew guide from the Community Ubuntu Documentation, again.

This adds Amavis-new, clamav and spamassassin and gets them working with postfix.

Installation

These are the main packages needed:

root@myserver:/home/steven# apt-get install amavisd-new spamassassin clamav-daemon

These packages provide better spam protection and better scanning of archives:

root@myserver:/home/steven# apt-get install libnet-dns-perl libmail-spf-query-perl pyzor razor
root@myserver:/home/steven# apt-get install arj bzip2 cabextract cpio file gzip lha nomarch pax rar unrar unzip unzoo zip zoo

These packages are optional, but will provide better protection.

Configuration

ClamAV

Add clamav to the amavis group and vice versa.

root@myserver:/home/steven# adduser clamav amavis
Adding user `clamav' to group `amavis' ...
Adding user clamav to group amavis
Done.
root@myserver:/home/steven# adduser amavis clamav
Adding user `amavis' to group `clamav' ...
Adding user amavis to group clamav
Done.

Spamassassin

Edit /etc/default/spamassassin and make sure these two settings are set:

  • ENABLED=1 (near the top)
  • CRON=1 (at the bottom)

Start spamassassin:

root@myserver:/home/steven# service spamassassin restart
Restarting SpamAssassin Mail Filter Daemon: spamd.

Amavis

Edit /etc/amavis/conf.d/15-content_filter_mode so it has this for its whole contents:

use strict;

# You can modify this file to re-enable SPAM checking through spamassassin
# and to re-enable antivirus checking.

#
# Default antivirus checking mode
# Uncomment the two lines below to enable it
#

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);


#
# Default SPAM checking mode
# Uncomment the two lines below to enable it
#

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

1;  # insure a defined return

Restart amavis:

root@myserver:/home/steven# service amavis restart
Stopping amavisd: amavisd-new.
Starting amavisd: amavisd-new.

Postfix

Add an option to /etc/postfix/main.cf using:

root@myserver:/home/steven# postconf -e 'content_filter = smtp-amavis:[127.0.0.1]:10024'

Add this to the end of /etc/postfix/master.cf:

smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20

127.0.0.1:10025 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=127.0.0.0/8
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

Find the line 'pickup' and add two lines to it so that it looks like this:

pickup    fifo  n       -       -       60      1       pickup
         -o content_filter=
         -o receive_override_options=no_header_body_checks

Reload postfix:

root@myserver:/home/steven# service postfix reload
 * Reloading Postfix configuration...             [ OK ]

That should have done it.

Testing

To check that things are working, let's use telnet, again:

root@myserver:/home/steven# telnet localhost 10024
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 [127.0.0.1] ESMTP amavisd-new service ready
quit
221 2.0.0 [127.0.0.1] amavisd-new closing transmission channel
Connection closed by foreign host.

You can check messages for these headers, too:

  • X-Spam-Level:
  • X-Virus-Scanned: Debian amavisd-new at mail.example.com
  • X-Spam-Status: No, hits=-2.3 tagged_above=-1000.0 required=5.0 tests=AWL, BAYES_00
  • X-Spam-Level:

Courier

This is a basic installation of Courier which will only deliver mail to users with user accounts.

Installation

E-mail can work just fine with this one package. Install this package:

root@myserver:/home/steven# apt-get install courier-imap-ssl

If you want IMAP without SSL, too, then add 'courier-imap' to that command.

If you want POP3 (ugh) then there are 'courier-pop-ssl' and 'courier-pop' packages.

Configuration

Add these directories to the skel directory so that they will be created by default for future users:

maildirmake /etc/skel/Maildir
maildirmake /etc/skel/Maildir/.Drafts
maildirmake /etc/skel/Maildir/.Sent
maildirmake /etc/skel/Maildir/.Trash
maildirmake /etc/skel/Maildir/.Templates

Copy and paste the whole block.

If you intend to use any existing users, they will need to have the directories created (change 'steven'):

root@myserver:/home/steven# cp -r /etc/skel/Maildir /home/steven/
root@myserver:/home/steven# chown -R steven:steven /home/steven/Maildir
root@myserver:/home/steven# chmod -R 700 /home/steven/Maildir

That should give a working system. Of course, you'll need to open up the firewall to connect from the internet.

Appendix

  1. SSH — ports to open
  2. Create your own SSL certificated for IMAPS

SSH

Mail ports

These can either be written as separate rule for fine-grained control, or lumped together.

The former:

# Allow inbound IMAPS
-A INPUT -i eth0 -p tcp --dport imaps -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport imaps -m state --state ESTABLISHED -j ACCEPT
# Allow inbound submission
-A INPUT -i eth0 -p tcp --dport submission -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport submission -m state --state ESTABLISHED -j ACCEPT
# Allow inbound SMTP
-A INPUT -i eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT
# Allow outbound SMTP
-A OUTPUT -o eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT

The latter:

# Allow inbound IMAPS, submission and SMTP
-A INPUT -i eth0 -p tcp -m multiport --dports imaps,submission,smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m multiport --sports imaps,submission,smtp -m state --state ESTABLISHED -j ACCEPT
# Allow outbound SMTP
-A OUTPUT -o eth0 -p tcp --dport smtp -m state --state NEW,ESTABLISHED -j ACCEPT
-A INPUT -i eth0 -p tcp --sport smtp -m state --state ESTABLISHED -j ACCEPT

Note: don't add both of these sets of rules. Only one is needed.

You can, of course, add other ports such as imap, pop3s and pop3.

IMAP-SSL certificates — creating your own

To create your own SSL certificates in place of the default courier ones:

root@myserver:/home/steven# openssl req -x509 -newkey rsa:4096 -keyout "/etc/ssl/private/courier-imapd-ssl.pem" -out "/etc/ssl/private/courier-imapd-ssl.pem" -nodes -days 1825
root@myserver:/home/steven# openssl req -new -outform PEM -out "/etc/ssl/private/courier-imapd-ssl.crt" -newkey rsa:4096 -nodes -keyout "/etc/ssl/preivate/courier-imapd-ssl.key" -keyform PEM -days 1825 -x509

Edit /etc/courier/imapd-ssl and change TLS_CERTFILE to /etc/ssl/private/courier-imapd-ssl.pem

Some housekeeping and restart the service:

root@myserver:/home/steven# chmod 640 /etc/ssl/private/courier-imapd-ssl.*
root@myserver:/home/steven# chgrp ssl-cert /etc/ssl/private/courier-imapd-ssl.*
root@myserver:/home/steven# service courier-imap-ssl restart
 * Stopping Courier IMAP-SSL server imapd-ssl     [ OK ]
 * Starting Courier IMAP-SSL server imapd-ssl     [ OK ]