Tuesday, June 19, 2012

Running a simple performance test

Now that we have our first recorded script, let's run a simple performance test of the Mongoose HTTP server.  I've provided a sample Grinder properties file in C:\Grinder\scripts\grinder.properties. The Grinder Agent knows how to find this file from the DOS environment variable %GRINDERPROPERTIES% set in C:\Grinder\setGrinderEnv.cmd.

In addition, theTCPProxy recorder was configured in C:\Grinder\startProxy.cmd to output its generated Grinder script to C:\Grinder\scripts\grinder_recorded.py. Rename grinder_recorded.py to grinder.py so it matches the name specified in grinder.properties file. The reason for renaming the file is to prevent accidentally overwriting of grinder.py if we decide to do a future re-recording.

Now start the Grinder Console using C:\Grinder\startConsole.cmd:

And then start the Grinder Agent using C:\Grinder\startAgent.cmd. You should see a DOS window showing the Agent starting up and attempted to connect to the Console:

The Agent is now waiting for instruction from the Console, so go to the Action menu and select Start processes. A pop-up window warning that a grinder.properties file has not been selected. You can ignore this for now, and click OK. Your performance test should start and if all is well you should see the TPS counter in the Console displaying the number of requests The Grinder is making to the HTTP server:

After a short while go to Action / Stop Processes to end the performance test. If the Console is reporting errors similar to the following:

then it might be the case that the HTTP server is not running. To confirm if this is the case, click here again to confirm the simple "Hello World" HTML page is being served. If you get an error in your browser, restart the HTTP server by clicking on C:\Grinder\startHttpd.cmd and then check again you can browse to the page. Restart the Agent which was stopped when you chose Stop Processes above, and then rerun the test. You can find out more about the other features of the Grinder Console in the Grinder Console documentation.

Now lets look at the log files that The Grinder generated in C:\Grinder\logs:

There are three log files that the Grinder could possibly create. The Grinder always should create a data file computername-0-data.log, which includes a line of statistics for each HTTP request that was made, such as response time and any client or server errors. If there were any errors during the test you should also see a file computername-0-error.log. This is where any errors are logged, should there have been any during the test.  Finally, the main log file should be created called computername-0.log which contains a log of actions each Grinder thread performed. You can also write custom messages to this log file using the grinder.logger.output facility. You can have a look at the very useful Script Gallery documentation for more examples on how to use this logging facility. The most interesting part of this file is the final statistics summary computed and logged at the end:

             Tests        Errors       Mean Test    Test Time    TPS          Mean         Response     Response     Mean time to Mean time to Mean time to 
                                       Time (ms)    Standard                  response     bytes per    errors       resolve host establish    first byte   
                                                    Deviation                 length       second                                 connection                

(Test 100    10163        0            15.45        5.32         313.82       0.00         0.00         0            0.00         0.00         0.00)         "Page 1"
Test 101     10163        0            15.20        5.16         313.82       45.00        14121.82     0            0.02         1.50         13.87         "GET /"

Totals       10163        0            15.20        5.16         313.82       45.00        14121.82     0            0.02         1.50         13.87        
             (10163)      (0)                                                                                                                               

These statistics are computed from the computername-0-data.log file and provide a summary of which tests were run, any errors, the Mean Test Time (also known as Response Time), average TPS, and lots of other useful summary statistics.

I hope that the last two posts provided a brief overview of recording and execution of a Grinder test script, and how to examine the results of the test run. Next time I plan to show how to develop a RESTful test script that can be used to test an API. This will not use the TCPProxy recording function to generate a Jython script, but instead be coded from scratch and incorporate some third-party Java libraries for parsing the REST responses.

Sunday, June 17, 2012

Setting Up The Grinder under Windows

For this first tutorial I'm going to use Windows rather than Linux, as it is likely that most Grinder users are more familiar with Windows than Linux. I've created a Zip file with a full installation of the latest version of The Grinder 3.9.1 (as of writing), in C:\Grinder. The set up follows The Grinder User Guide closely, using standard Windows .cmd files. It also includes a lightweight HTTP server called "Mongoose" that supports server-side PHP, which I plan to use to develop more interesting test scripts in the future.

You can download the ZIP file from here. The only thing the Zip file does not contain is the Sun Java JRE. For the purposes of the tutorial we're going to use C:\Grinder as The Grinder installation folder. I'm going to assume you have already installed the Sun JRE and that Java's home (JAVA_HOME)  is C:\Windows\System32. If this is not the case you will have to change the C:\Grinder\setGrinderEnv.cmd command file to point to your JAVA_HOME.

Once you have everything downloaded, unzipped, and installed. Double-click on the command file C:\Grinder\startHttpd.cmd to start the Mongoose HTTP server on port 8080. A DOS command window should appear similar to the following, indicating that the Mongoose HTTP server is running:

You should then be able to see the "Hello World" page by going to the Mongoose root here. In Firefox you should hopefully see something similar to the following:

Note that the HTTP server tries to listen on port 8080. You may have to tell Windows Firewall to allow it to access the port, and also note that this is a development HTTP server that shouldn't be left running as it is likely very insecure. Only run it when you are working through the tutorial and when your PC is not on a public network. Email me at flying kiwi guy @ gmail.com (no spaces) if you have any problems.

The next step is to run the TCPProxy. The TCPProxy is a program that records each click you do when using Firefox to browse a website. The recording of clicks (actually HTTP requests) will be translated into Jython code that is directly executable by The Grinder. Double-click on  C:\Grinder\setGrinderEnv.cmd and you should see a the TCPProxy start up in a second DOS command window:

You will also see the TCPProxy Console window, similar to the following:

Note that the TCPProxy is configured to listen on port 8001. You need to change your browser settings in order for the TCPProxy to intercept ("proxy") the client HTTP requests to the HTTP server. In Firefox, choose Tools -> Options -> Advanced:

Then click on the Network tab and then Settings... and make the following changes:

You should confirm that the "No Proxy For:" is blank. Our HTTP server is running on on localhost:8080, and the TCPProxy is running on localhost:8001 ( is the IP address for localhost). We want to proxy any Firefox request through the TCPProxy at localhost:8001. Let's see if we can now still access our Hello World by again clicking here. Not very exciting, but you should again see the same page:

What is different this time is that the TCPProxy should have recorded your Firefox click, and maybe some clicks to this blog too. Let's check. Click the Stop button in the TCPProxy Console window. The TCPProxy should have created the file C:\Grinder\scripts\grinder_recorded.py like so:

Open up this file and you should see some automatically generated code similar to the following:

# The Grinder 3.9.1
# HTTP script recorded by TCPProxy at 17-Jun-2012 16:39:01

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()

# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)

# These definitions at the top level of the file are evaluated once,
# when the worker process is started.

url0 = 'http://localhost:8080'

# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0)
request101 = Test(101, 'GET /').wrap(request101)

class TestRunner:
  """A TestRunner instance is created for each worker thread."""

  # A method for each recorded page.
  def page1(self):
    """GET / (request 101)."""
    result = request101.GET('/', None,
      ( NVPair('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20100101 Firefox/12.0'),
        NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
        NVPair('Accept-Language', 'en-us,en;q=0.5'),
        NVPair('Accept-Encoding', 'gzip, deflate'),
        NVPair('If-Modified-Since', 'Sun, 17 Jun 2012 15:25:01 GMT'),
        NVPair('If-None-Match', '\"4fddf6cd.2d\"'), ))

    return result

  def __call__(self):
    """Called for every run performed by the worker thread."""
    self.page1()      # GET / (request 101)

def instrumentMethod(test, method_name, c=TestRunner):
  """Instrument a method with the given Test."""
  unadorned = getattr(c, method_name)
  import new
  method = new.instancemethod(test.wrap(unadorned), None, c)
  setattr(c, method_name, method)

# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')

In the next post I will cover how to now execute your recorded script within The Grinder and obtain some performance test results on how the Mongoose HTTP server performs when it is serving our simple HTML page.

Remember, don't forget to disable the proxy settings in Firefox, if you plan to use Firefox for normal browsing! You might want to check out a useful Firefox plugin that can be used to easily change your proxy settings.

Introduction to Agile Grinding

Hello everyone - this is hopefully one of many future posts exploring the features of the performance test suite called The Grinder. I've used a lot of different performance test tools in my career, and I've come to realise that The Grinder provides an elegant balance between functionality, usability, and flexibility. It doesn't have all the bells and whistles of an enterprise performance test tool such as HP LoadRunner or Borland SilkPerformer, but then again it also doesn't have the five or six figure price tag. What it does have is a very rich set of features out of the box and, using the embedded Jython interpreter, a comprehensive and easily extensible set of programming tools that can leverage all of the power of Java directly from within Jython and The Grinder. Need a JSON interpreter? Simply import the Codehaus Jackson JSON stream reader. Need to integrate with ActiveMQ? Simply import the ActiveMQ client libraries and start processing messages from a JMX queue.

At the same time the basic workflow of creating and running test scripts is easily learnable by non-specialist programmers. Once you get your head around the Python/Jython syntax you'll find it hard to go back to languages such as Java or C#. For a start, you'll forget to append ";" and use "{}" instead of indentation! Writing Jython makes for clean, succinct scripts, and it is a pleasure to "hit the button" and see the script spin up within The Grinder framework and start producing hands-on, relevant, and reliable performance test results.

My background for over 18 years is in Linux systems administration, performance testing, and performance analysis. I'm going to start with a general introductory tutorial covering all the main features of The Grinder, and then start to develop a framework of Open Source tools around The Grinder that focuses on Agile Methods. I have noticed that almost all areas of software development are adopting agile principles in part or in full. This includes functional testing, with such paradigms as Test Driven Development. However, in my experience performance testing still seems to operate in a very "once it is completely developed, then we'll do a complete performance test" waterfall approach. This needs to change, as performance testing needs to become more flexible and provide much earlier feedback so it can better integrate into agile methodologies. I don't see the commercial enterprise test tools being capable of entering this area, as the intent is small, fast, simple iterative feedback that is directly relevant to the developers cutting the code. Too often performance test results are remote in time and space from the very people who need to understand and respond to them, namely the developers who need to know immediately: "if I check-in this one line change, what will be the performance implications?".