Add Configuration Test to Logstash Running As A Service

When running Logstash as a service, there is no option to run the --configtest flag. Here's how to add it to the logstash service.

When making changes to the logstash config, if you were to start the Logstash service using service logstash start, the service will start up (it takes a while to do so) and then crash silently if there are errors in the configuration files. It is possible to check the config files beforehand and get a verbose output by using the --configtest flag when starting logstash.

Service Method

Find the logstash service file at /etc/init.d/logstash and add the function configtest after the force_stop function and change the case switch:

force_stop() {
  if status ; then
    status && kill -KILL `cat "$pidfile"`

configtest() {
  args="$args --configtest"
  nice -n ${LS_NICE} chroot --userspec $LS_USER:$LS_GROUP / sh -c "
    cd $LS_HOME
    exec \"$program\" $args

case "$1" in
    if [ $code -eq 0 ]; then
      echo "$name is already running"
    exit $code
  stop) stop ;;
  force-stop) force_stop ;;
    if [ $code -eq 0 ] ; then
      echo "$name is running"
      echo "$name is not running"
    exit $code

    stop && start
    echo "Usage: $SCRIPTNAME {start|stop|force-stop|status|restart|configtest}" >&2
    exit 3

exit $?

This will allow you to run a configuration test without starting the logstash program. Use service logstash configtest to run a config test. It takes a while, so be patient. You might need to prefix it with sudo.

Alternative Method — Direct Run

For completeness, it is possible to run the logstash binary directly to run a config test. The method above simplifies the process.

Use this command to run it directly (you may need to prefix it with sudo):

/opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest


Organising Wowza Logs with Logstash and Grok

Pre-processing Wowza Streaming Engine (or Media Server) logs with Grok patterns syntax for better parsing with logstash.

>>>Click here to jump to the pattern to Copy & Paste.<<<

In a previous blog post I described setting up Logstash with Elasticsearch, Kibana and logstash-forwarder (lumberjack). In my logstash config file, I referenced a grok pattern for parsing a Wowza access log. I will describe that pattern, here.

This was created for Wowza Media Server 3.6 and works with Wowza Streaming Engine 4.0.


Using the grok filter in Logstash allows for patterns to be captured from a logstash field. These captured patterns can then become fully-fledged fields. Instead of a field called message containing the whole Wowza log entry, it can be expanded so that each part of the message becomes a field.

A Wowza log entry looks like this:

2014-06-11\t06:53:34\tUTC\tcreate\tstream\tINFO\t200\t-\t-\t_defaultVHost_\tvod\t_definst_\t0.0\tstream.example.com\t1935\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4?session_token=xxxxx&username=steven\t219.92.86.40\trtsp\t-\tSonyC6603 Build/10.5.A.0.230 stagefright/1.2 (Linux;Android 4.4.2)\t1335194632\t0\t0\t3\t0\t0\t0\tamazons3/my-bucket/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven\t-\t-\t-\t-\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4?session_token=xxxxx&username=steven\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven\trtsp://stream.example.com:1935/vod/_definst_/mp4:FootballTV/450kbps/BPLClipOfTheWeekArsenalVsFulham_450kbps.mp4\tsession_token=xxxxx&username=steven

It's not very useful like this. I can't easily search through it. I certainly can't see information at a glance. Each of these tab-delimited fields means something. These "columns" have names which are specified in the user manual.

With grok, we can create new fields using the names in the Wowza manual and the values in the message field. Instead of a single message field, we can have:




















...and so on. This is much better.

Basic Patterns

Creating grok patterns is described in the logstash filters documentation.


The default grok date formats do not include the YYYY-MM-DD format which Wowza uses. We can use the default year, month number and day number.


This gives a syntax called YMDDATE.


The fields in the log are separated by tab characters by default. Anything after the date and time AND starts with a tab character is a new field.


This becomes a problem when the xComment field can be written to in your own Wowza Java modules through WMSLogger.info() and similar methods. As that field can contain user data (things not written by Wowza), then it is not unreasonable to expect tab characters in that field.

By that reasoning, I added another, special capture after the xComment capture:


This prevents tab characters in xComment from causing the grok pattern to break and start the middle of the xComment as other fields. It adds the offending part of the comment to a new field called xCommentMalformed. The field can have any unique name.

Specific fields

There are fields in the log which could be captured by the grok syntax defaults, such as IP, PATH and NONNEGINT. These cannot be used because Wowza will replace the values in these fields with a hyphen (-) if they were not populated by the message.

To address this issue, I created some specific implementations of these syntax labels.

WOWZIP (?:%{IP}|-)

The Full Pattern

The part you probably just wanted to copy and paste.

This creates a syntax named WOWZAACCESSLOG.

This pattern is for Wowza Streaming Engine 4.1.

WOWZIP (?:%{IP}|-)
WOWZAACCESSLOG %{YMDDATE:date}\t%{TIME:time}\t%{TZ:tz}\t%{WOWZENTRY:xEvent}\t%{WORD:xCategory}\t%{LOGLEVEL:xSeverity}\t%{WOWZNONNEGINT:xStatus}\t%{WOWZENTRY:xCtx}\t%{WOWZENTRY:xComment}(?<xcommentmalformed>\t.*)?\t%{WOWZENTRY:xVhost}\t%{WOWZENTRY:xApp}\t%{WOWZENTRY:xAppinst}\t%{NUMBER:xDuration}\t%{WOWZENTRY:sIp}\t%{WOWZNONNEGINT:sPort}\t%{WOWZENTRY:sUri}\t%{WOWZIP:cIp}\t%{WOWZENTRY:cProto}\t%{WOWZENTRY:cReferrer}\t%{WOWZENTRY:cUserAgent}\t%{WOWZENTRY:cClientId}\t%{WOWZNONNEGINT:csBytes}\t%{WOWZNONNEGINT:scBytes}\t%{WOWZENTRY:xStreamId}\t%{WOWZNONNEGINT:xSpos}\t%{WOWZNONNEGINT:csStreamBytes}\t%{WOWZNONNEGINT:scStreamBytes}\t%{WOWZENTRY:xSname}\t%{WOWZENTRY:xSnameQuery}\t%{WOWZPATH:xFileName}\t%{WOWZENTRY:xFileExt}\t%{WOWZNONNEGINT:xFileSize}\t%{WOWZNUMBER:xFileLength}\t%{WOWZENTRY:xSuri}\t%{WOWZENTRY:xSuriStem}\t%{WOWZENTRY:xSuriQuery}\t%{WOWZENTRY:csUriStem}\t%{WOWZENTRY:csUriQuery}

This pattern is for older versions of Wowza Streaming Engine and Wowza Media Server. If a pattern gives you _grokparsefailure tags or other errors, try the other.

WOWZIP (?:%{IP}|-)
WOWZAACCESSLOG %{YMDDATE:date}\t%{TIME:time}\t%{TZ:tz}\t%{WOWZENTRY:xEvent}\t%{WORD:xCategory}\t%{LOGLEVEL:xSeverity}\t%{WOWZNONNEGINT:xStatus}\t%{WOWZENTRY:xCtx}\t%{WOWZENTRY:xComment}(?<xCommentMalformed>\t.*)?\t%{WOWZENTRY:xVhost}\t%{WOWZENTRY:xApp}\t%{WOWZENTRY:xAppinst}\t%{NUMBER:xDuration}\t%{WOWZENTRY:sIp}\t%{WOWZNONNEGINT:sPort}\t%{WOWZENTRY:sUri}\t%{WOWZIP:cIp}\t%{WOWZENTRY:cProto}\t%{WOWZENTRY:cReferrer}\t%{WOWZENTRY:cUserAgent}\t%{WOWZENTRY:cClientId}\t%{WOWZNONNEGINT:csBytes}\t%{WOWZNONNEGINT:scBytes}\t%{WOWZENTRY:xStreamId}\t%{WOWZNONNEGINT:xSpos}\t%{WOWZNONNEGINT:csStreamBytes}\t%{WOWZNONNEGINT:scStreamBytes}\t%{WOWZENTRY:xSname}\t%{WOWZENTRY:xSnameQuery}\t%{WOWZPATH:xFileName}\t%{WOWZENTRY:xFileExt}\t%{WOWZNONNEGINT:xFileSize}\t%{WOWZNONNEGINT:xFileLength}\t%{WOWZENTRY:xSuri}\t%{WOWZENTRY:xSuriStem}\t%{WOWZENTRY:xSuriQuery}\t%{WOWZENTRY:csUriStem}\t%{WOWZENTRY:csUriQuery}

The difference is that x-file-length can be a decimal in newer logs, whereas my old pattern only looks for positive integers.

It's worth noting that these patterns work if your logging fields are in this order:

  1. date
  2. time
  3. tz
  4. x-event
  5. x-category
  6. x-severity
  7. x-statusx-ctx
  8. x-comment
  9. x-vhost
  10. x-app
  11. x-appinst
  12. x-duration
  13. s-ip
  14. s-port
  15. s-uri
  16. c-ip
  17. c-proto
  18. c-referrer
  19. c-user-agent
  20. c-client-id
  21. cs-bytes
  22. sc-bytes
  23. x-stream-id
  24. x-spos
  25. cs-stream-bytes
  26. sc-stream-bytes
  27. x-sname
  28. x-sname-query
  29. x-file-name
  30. x-file-ext
  31. x-file-size
  32. x-file-length
  33. x-suri
  34. x-suri-stem
  35. x-suri-query
  36. cs-uri-stem
  37. cs-uri-query

Using The Pattern In Logstash

I recommend reading a previous blog post for setting up logstash if you haven't already.

Save these patterns to a file. It can be any file as long as you remember the path and that logstash has permission to read it. Try /etc/logstash/patterns/wowza.

Edit your logstash configuration file /etc/logstash/conf.d/logstash.conf. Add a new grok filter:

filter {
    ... other filters ...
    grok {
        patterns_dir => "/etc/logstash/patterns"
        match => [ "message", "%{WOWZAACCESSLOG}" ]

Telling logstash the time

Logstash will use the current time to stamp the log entry. In order to tell logstash to use the time in the log message instead, it needs to be told where to find it.

The required date fields are date, time and tz as defined in the WOWZACCESSLOG grok pattern.

Create a new field containing a timestamp (named datetime) using grok's add_field function and then use the date logstash filter to capture it:

grok {
    patterns_dir => "/etc/logstash/patterns"
    match => [ "message", "%{WOWZAACCESSLOG}" ]
    add_field => [ "datetime", "%{date} %{time} %{tz}" ]
date {
    match => [ "datetime", "yyyy-MM-dd HH:mm:ss Z" ]


This post has covered the creation of custom grok syntax to parse a Wowza access log entry for use with logstash. It will separate the log parts into individual fields. This should make your Wowza logs considerably more useful and search-able, particularly when used with Elasticsearch.

When used in conjuction with my post covering the setting up of Elasticsearch, Logstash, Kibana and logstash-forwarder, these posts should give enough information to allow you to organise your Wowza logs.

Happy Wowza debugging!

P.S. On the subject of debugging, check out my post for debugging your Wowza modules code by stepping through the Java code in NetBeans (or any JIT debugger) remotely.


JIT debugging compiled Java on Wowza Streaming Engine with NetBeans

How to debug compiled Java code using NetBeans IDE to step through code and how to set-up Wowza Streaming Engine for just-in-time debugging.

In my daily work, I have a Maven project creating Wowza modules. Wowza's software uses reflection to invoke the methods in custom modules. This means that any exceptions which are thrown in these methods are wrapped in Java's java.lang.reflect.InvocationTargetException. Wowza Streaming Engine prints an error message to its logs, but only the message of this exception. The cause — that which is really the problem — is not reported. Anywhere.

To find out what was going on, I needed to use a debugger to step though the code, line by line, to identify the problem code.

On my development machine using NetBeans, I could not identify any problems. The issues only occurred on the test Wowza Streaming Engine, where the code was in its compiled, JAR form.

Here's how I went about using NetBeans on my development machine to debug the compile code "in production" as it was running.

Setting Up Maven to Compile for Debugging

Skip this step if you aren't using Maven.

In your project's pom.xml file, find or create the element project/build/plugins. Find or create the plugin whose artifactId is maven-compiler-plugin. Add configuration options so that it looks something like this:


The lines of note are 9, 10 and 11: compilerArgument, debug and debuglevel. These will tell Maven to compile the code and to include the debug symbols, which our debugger will need.

Compiling For Debugging Without Maven

Skip this step if you are using Maven.

You must tell the Java compiler to include the debug symbols when it compiles your code.

Simply supply the switch -g to the compiler.

Setting up Java for Debugging

Java needs to be run with certain switches to tell it to allow for remote debuggers to attach to it. What these switches are depends on the version of Java which is running. You can find this out by running this command:

java -version

The commands to allow remote debugging in the different versions of Java are:

JDK 1.3 and earlier
-Xnoagent -Djava.compiler=NONE -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6006
JDK 1.4
-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=6006
Newer JDKs

These are taken from a StackOverflow answer. I have not tried the switches for the earlier versions. If you are needing those, you should consider updating your JDK.

suspend tells Java to pause execution every time the program runs. You probably don't want this and actually want the program to run as normal until you stop it in the debugger. In that case, the argument is n.

address tells Java to listen on this port for attached debuggers. Remember this port as you'll need it later in NetBeans.

To get Wowza to start with these switches, there is a file containing the start-up switches for Java. The XML file is /usr/local/WowzaStreamingEngine/conf/Tune.xml. Add a VMOption to the /Root/Tune/VMOptions element with the switches which your version of Java requires. It should look something like:

  <!-- <VMOption>-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="${com.wowza.wms.AppHome}/logs"</VMOption> -->
  <!-- <VMOption>-Duser.language=en -Duser.country=US -Dfile.encoding=Cp1252</VMOption> -->
  <!-- <VMOption>-verbose:gc -Xloggc:"${com.wowza.wms.AppHome}/logs/gc_${com.wowza.wms.StartupDateTime}.log" -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime</VMOption> -->

The next time that Wowza is started, you should be able to see that Java is listening to port 6006 (or whichever port you chose).

Debugging in NetBeans

This example is using NetBeans IDE 7.4.

In the IDE, open the project which you wish to debug. That this code is on your development machine and the compiled code is on a different server does not matter.

Click on the Debug (shortcut Alt+D) menu item and then Attach Debugger...

Check the values in the Attach pop-up:

Debugger: Java Debugger (JPDA)
Connectior: SocketAttach (Attaches by socket to other VMs)
Transport: dt_socket
Port 6006
Timeout [ms]:

Change the Host and Port to point to the server which you want to debug.

It will look something like this:

NetBeans 7.4 Linux Attach Remote Debugger config

After you click on OK, NetBeans will attempt to connect to the Java process on the remote server. Once it is done, the Attaching Debugger progress bar at the bottom will eventually disappear and the standard debugging windows and buttons will be available. NetBeans should look like you are debugging local code.

You can now add breakpoints in real-time. When the executing Java on the remote server reaches these points in its own code, the execution will stop and you can step through the code and it will even evaluate the variables for you.


This guide should have allowed you to debug your compiled Java code remotely and to set-up a Wowza Streaming Engine for debugging.

It may be possible to use this guide as a base for debugging other Java programs.

When I was debugging the cause of my InvocationTargetException errors, I found that try-catch on Exception was not enough to catch the exceptions. I changed this to Throwable, which allowed me to catch everything. Don't leave your code in production catching Throwables; it is not intended for that. Stick with Exception.

Don't forget to switch off debug symbols for production code.