May 282011
 

This post shows how to render a list of product objects with associated category object using the webframeworks Rails, Wicket, Grails, Play, Tapestry, Lift, Context and JSP/Servlets. I’ve also benchmarked the response times.

INTRODUCTION

More »

The response time of websites is important, not just for the users, but also for SEO (Search Engine Optimalization) of a website. According to Google:

Speeding up websites is important — not just to site owners, but to all Internet users. Faster sites create happy users and we’ve seen in our internal studies that when a site responds slowly, visitors spend less time there. But faster sites don’t just improve user experience; recent data shows that improving site speed also reduces operating costs. Like us, our users place a lot of value in speed — that’s why we’ve decided to take site speed into account in our search rankings. We use a variety of sources to determine the speed of a site relative to other sites.

The response time of a website is the time between the HTTP request send from the webbrowser to the webserver and the corresponding HTTP reponse send from the webserver back to the browser.
There are many factors that determine this response time, such as geographical distance, Internet connection, the speed of the computer the browser is running on, and the performance of the webserver. The webservers performance is usually determined by database queries and the amount of data that has be to send back. But I was wondering how much the rendering performance of webframeworks impact the response time and how they compare to eachother.

In this post I’ll show how each you can render a list of products using each framework. This can be handy if you need a quick look at a framework, and see how it compares codewise to others. For each framework, I’ve measured the response times against the number of concurrent users and the number of product objects to render. I’ve also looked quickly at the memory consumption of each framework.

Please note, I’m not an expert in these frameworks, but I tried to use the most common solutions to render objects.
Also note that this post won’t show which framework is faster in general, it will only tell you which one is faster under the given environment/conditions and the given test case.
Another difficulty is that some frameworks have builtin features that shouldn’t be tested, like automatically opening database connection on each request. Some frameworks might have builtin security features that will increase the response times, or some might have builtin caching enabled by default. I haven’t looked at all these extra features that shouldn’t be part of the test case, so the results will only give a coarse indication of how fast a framework is in the given environment/test-case.

METHODS

More »

Test case

The test only consist of one page that is tested. This page contains a 2 column HTML layout using div tags. One column is the sidebar, the other the content area. The content area needs to render a list of products with it’s associated categories in a HTML table. The page does not contain components that need to be stateful (like forms etc.).
See GitHub for an example.

Each Product (class) contains the following properties:

-name: String
-price: Integer
-description: String
-categories: Hash-based set of Category

Each Category (class) contains the following properties:

-name: String

All solutions use an (static) in-memory List to retrieve products, so no database access was needed or set.

Framework features

For each framework, the page needs to use the following framework features:
-rendering of a list of products, each rendered as a custom component so that the layout of each product can be reused on other pages. For each product, it’s associated categories are shown.
-Separation of the page into a template that can also be used for other pages, and the content that is unique for each page (this content is surrounded by the template).
-Injection of the page name (including the number of products) in the title tag, and in the master template page name.

Measurements

The following variables are used in the response time measurements:
-Number of concurrent users
-Number of products

The memory usage is measured only for the 16 concurrent user with 1000 products test. Usage is measered by roughly looking at the average values using Windows Task Manager. CPU utilization is also measured in this way.

Each framework should return approximately the same HTML page size. Differences can occure because the way I wrote the template pages. For example: there is a difference in size between:

<g:each var="cat" in="${product.categories}">$${cat.name}, </g:each>

and

<g:each var="cat" in="${product.categories}">
	$${cat.name},
</g:each>

I measured the page sizes to make sure that there weren’t big differences. Big differences could impact performance.

The HTTP benchmark was performed by the tool Apache JMeter 2.4, which also measured the response times. Within JMeter, a Response Assertion was used to make sure that the correct page was returned (and not some error pages). The Response Assertion tested whether the result contained the string “Company” (which is part of the layout). No sessions/cookies nor HTTP request managers were set in JMeter, so each HTTP request was performed independently of the other requests.

Test system

-OS: Windows 7 Professional SP1 64bit
-CPU: AMD Phenom II X4 955
-MEM: 4 GB RAM
-Java: SE Update 24, 32bit
-Webserver for JVM based frameworks: apache-tomcat-7.0.12-windows-x86, run with -Xmx1024m

Tomcat was set to Xmx 1024mb, which gives the Java JVM at most 1024mb of available RAM to use. This should be enough to minimize garbage collection.

When settting Tomcat environment options under Windows, don’t use quotes!
set JAVA_OPTS=”-Xmx1024m”
won’t work, even though it is used in books like “Tomcat – The Definitive Guide”. Use instead:
set JAVA_OPTS=-Xmx1024m
You can verify this in the “Tomcat Web Application Manager” under “Server Status”, look for “Max memory”.

Framework: JSP, JSTL, Servlets

Specs:
-JSP 2.1
-JSTL 1.2
-Servlet 2.5

More »

file: WEB-INF/web.xml (URL mappings)
....
<servlet>
	<servlet-name>products</servlet-name>
	<servlet-class>www.ProductsServlet</servlet-class>
</servlet>

<servlet-mapping>
	<servlet-name>products</servlet-name>
	<url-pattern>/products</url-pattern>
</servlet-mapping>
.....

web.xml is used to map the uri /products to the ProductsServlet class, which acts as a MVC controller.

file: WEB-INF/classes/www/ProductsServlet.java (controller)
public class ProductsServlet extends HttpServlet {
	@Override
	protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
		request.setAttribute("products", Service.getProducts());
		
		RequestDispatcher view = request.getRequestDispatcher("/products.jsp");
		view.forward(request, response);
	}	
}

The Servlet container (Tomcat) sends incoming HTTP requests to ProductsServlet (MVC controller), which retrieves the product data as a List object, and puts it in the request object as attributes and dispatches/forwards/pushes the request to products.jsp (MVC view).

file: products.jsp (view)
<%@ page language="java" contentType="text/html; charset=ISO-8859-1" pageEncoding="ISO-8859-1"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@ taglib prefix="company" tagdir="/WEB-INF/tags" %>
<%@ taglib prefix="company2" uri="company" %>

<company:header title="Products Listing" />
					
<table>
	<c:forEach var="product" items="${products}">
		<tr>
			<td><company:product product="${product}" /></td>
			<td>
				<c:forEach var="cat" items="${product.categories}">
					${cat.name}, 
				</c:forEach>
			</td>
		</tr>
	</c:forEach>
</table>

<company:footer />

Products.jsp is the view, which receives data pushed from the ProductsServlet (controller).
The taglib lines define the location of JSP based tags.
The c:forEach tags are used to loop over the products and categories. The “company:product” tag is used as a custom component, which can be implemented as a JSP Tag File (prefix “company”), or as a TLD (Tag Library Descriptor) component (prefix “company2″). The company:header tag and company:footer tag are implemented as Tag Files, which surround the page content with the default layout that can be shared among other pages.

file: WEB-INF/tags/header.tag (heading template)
<%@ attribute name="title" required="true" rtexprvalue="true" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="default.css" media="screen" rel="stylesheet" type="text/css" />
		
		<!-- Title needs to be injected -->
		<title>${title}</title>
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<!-- Page name needs to be injected -->
					<div><h2>${title}</h2></div>
					<div class="innertube">

This Tag File is the first part of the default layout that can be shared among other pages. The first line defines data that can be passed to this Tag File.

(The footer heading template is similar to this file, but now shown here.)

file: WEB-INF/tags/product.tag (JSP based component)
<%@ tag body-content="empty" %>
<%@ attribute name="product" required="true" rtexprvalue="true" type="domain.Product" %>
<div class="product">
	<img src="${product.name}.jpg" />
	<span class="productname">${product.name}</span>, <span class="price">$${product.price}</span>
</div>

This is the Tag File based custom component used to render products. The attribute tg is used to define the parameters that can be passed to this file.

An alternative to the Tag File based custom component is the use of TLD (Tag Library Descriptor) + Java based code. This alternative isn’t tested in this post, but it’s probably faster than the Tag File based custom component. The code of this alternative is shown below.

(alternative, not tested) file: WEB-INF/default.tld (component declaration)
<?xml version="1.0" encoding="UTF-8"?>
<taglib xmlns="http://java.sun.com/xml/ns/j2ee" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee 
	web-jsptaglibrary_2_0.xsd" version="2.0">
	<tlib-version>1.0</tlib-version>
	<uri>company</uri>
	<tag>
		<name>product</name>
		<tag-class>www.ProductTag</tag-class>
		<body-content>empty</body-content>
		
		<attribute>
			<name>product</name>
			<required>true</required>
			<rtexprvalue>true</rtexprvalue>
		</attribute>
	</tag>
</taglib>

This is the default TLD (Tag Library Descriptor), which declares the company2:product tag used in products.jsp. It references to the www.ProductTag Java class file.

(alternative, not tested) file: WEB-INF/classes/www/ProductTag.java (Java based component)
public class ProductTag extends SimpleTagSupport {
	private Product product;
	
	public void setProduct(Product product) {
		this.product = product;
	}
	
	@Override
	public void doTag() throws JspException, IOException {
		JspWriter w = getJspContext().getOut();
		w.println("<div class=\"product\">");
		w.println("<img src=\"" + product.getName() + ".jpg\" />");
		w.println("<span class=\"productname\">" + product.getName() + "</span>,");
		w.println("<span class=\"price\">$" + product.getPrice() + "</span></div>");
	}
}

This writes a product as a custom component, it matches the TLD.

Framework: Wicket

Specs:
-Wicket 1.5-RC3

Because of a bug ( https://issues.apache.org/jira/browse/WICKET-3740 ) in Wicket, which might impact performance, Wicket is also tested for the latest snapshot version of Wicket from trunk for 1.5-RC3. (Called “Wicket-trunk” in the test results.)

More »

file: wicketapp/WicketApplication.java
public class WicketApplication extends WebApplication {    	
	...
	@Override
	public void init() {
		...
		mountPage("products", ProductsPage.class);
	}
}

The mountPage method maps the URI /products to the class ProductsPage.

file: wicketapp.TemplatePage.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="default.css" media="screen" rel="stylesheet" type="text/css" />
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<div><h2 wicket:id="pageName">Producs listing</h2></div>
					<div class="innertube">
					
						<wicket:child />
						
					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

This defines a Wicket page that can be used as a template for other Wicket pages; it serves as a base class for those other pages.

This is the template that is shared by other pages (subclasses). Together with TemplatePage.java, TemplatePage.html forms a single page.
The wicket:id is used to bind the HTML tag to a Wicket component. This allows TemplatePage.java to access and modify the HTML element/component, including it’s body. In this case, the wicket:id named “pageName” is used to allow TemplatePage.java to set the name of the page.

The wicket:child element is used to define the place where the content of subclass pages (such as the ProductsPage) will be shown.

file: wicketapp/TemplatePage.java
public class TemplatePage extends WebPage {
	
	private Model pageNameModel = new Model();
	
    public TemplatePage(final PageParameters parameters) {
    	// h2 page name
    	add(new Label("pageName", pageNameModel));
    }
    
    public void setPageName(String name) {
    	pageNameModel.setObject(name);
    }    
}

This is the Java code that is associated with TemplatePage.html. It adds a Label component to the page, which is passed the wicket:id value from TemplatePage.html, and a model object.
Wicket uses models to attach data to components. In case of the Label component is given a model object that is able to retrieve the name of the page.

These models are required because Wicket is designed to let pages be stateful and in Wicket the retrieval of data for a component if seperated from the component itself. This allows pages to be serialized without requiring all the attached model data to be serialized.

file: wicketapp/ProductsPage.html
<wicket:head>
	<title>Product Listing</title>
</wicket:head>

<wicket:extend>
	<table>
		<tr wicket:id="products">
			<td>
				<div wicket:id="product" />
			</td>
			<td>
				<wicket:container wicket:id="categories">
					<span wicket:id="category"></span>, 
				</wicket:container>
			</td>
		</tr>
	</table>
</wicket:extend>

ProductsPage.java extends TemplatePage.java, this will cause ProductsPage.html’s wicket:extend body to be placed at TemplatePage.html’s wicket:child tag’s location.
The wicket:id tags are used to bind the HTML tags to a Wicket component. This allows ProductsPage.java to access and modify those HTML elements/components, including the bodies.
The wicket:container tag is used as a grouping/container, which is bound to a repeating view component that is used to iterate over the categories.
The wicket:head tags allows child pages and components to add tags to the HTML head.

file: wicketapp/ProductsPage.java
public class ProductsPage extends TemplatePage {
	public ProductsPage(final PageParameters parameters) {
		super(parameters);
		ListView<Product> productsLV = new ListView<Product>("products", Service.getProducts()) {
			@Override
			protected void populateItem(ListItem<Product> li) {
				Product product = li.getModelObject();
				ProductPanel productPanel = new ProductPanel("product", product);
				li.add(productPanel);
				
				List<Category> categories = new ArrayList<Category>(product.getCategories());
				
				ListView<Category> categoriesLV = new ListView<Category>("categories", categories) {
					@Override
					protected void populateItem(ListItem<Category> catLi) {
						Category category = catLi.getModelObject();
						catLi.add(new Label("category", category.getName()).setRenderBodyOnly(true));
					}
				};
				li.add(categoriesLV);
			}
		};
		add(productsLV);		
	}
}

ProductsPage.java created the products page together with ProductsPage.html. It is a subclass of TemplatePage (which defines the template that is shared by other pages). It has a ListView componenent which iterates over the products. Each iteration is defined in the populateItem method. Each iteration creates a ProductPanel (a custom component) and a label containing the categories of each product. A nested LiewView is used to iterate over the categories. The strings such as “product”, “categories” and “category” are the component instance names, which match the wicket:id tag values used in ProductsPage.html.
Wicket uses models to attach data, e.g. a list of products, to components. In case of the Label component, the category.getName() string is automatically wrapped in a model.
These models are required because Wicket is designed to let pages be stateful and in Wicket the retrieval of data for a component if seperated from the component itself. This allows pages to be serialized without requiring all the attached model data to be serialized.

file: wicketapp/ProductPanel.html
<wicket:panel>
	<!-- This part/layout of each product needs to be made as a custom component, so that it can be reused at other pages -->
	<div class="product">
		<img wicket:id="image" />
		<span wicket:id="productname" class="productname">A name of product1....</span>, <span wicket:id="price" class="price">$150</span>
	</div>
</wicket:panel>

This is the layout of the custom product component. The wicket:id tags give HTML elements unique names, so that the HTML element can be accessed/modified/replaced as a Wicket component in ProductPanel.java.

file: wicketapp/ProductPanel.java
public class ProductPanel extends Panel {

	public ProductPanel(String id, Product product) {
		super(id);
			
		WebComponent img = new WebComponent("image");
		String imgUrl = product.getName() + ".jpg";
		img.add(new SimpleAttributeModifier("src", imgUrl));
		add(img);
		
		Label nameLabel = new Label("productname", product.getName());
		add(nameLabel);
		
		Label priceLabel = new Label("price", product.getPrice().toString());
		add(priceLabel);
	}
}

This is the Java code of the custom product component. This custom component is composed of three child Wicket components, each consisting of a wicket:id name and associated product data.
The SimpleAttributeModifier is used to add the src HTML attribute of the img HTML element in ProductPanel.html.

production settings

set to production mode using:
System.setProperty(“wicket.configuration”, “deployment”);

Framework: Grails

Specs:
-Grails 1.3.7
-Grails 1.4.0.M1

More »

file: views/layouts/main.gsp
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="${createLinkTo(dir:'', file:'default.css')}" media="screen" rel="stylesheet" type="text/css" />
		<title><g:layoutTitle default="Grails" /></title>
        <g:layoutHead />
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<div><h2>${pageName}</h2></div>
					<div class="innertube">
					
						<g:layoutBody />
						
					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

This defines the default layout that can be shared among other pages. The g:layoutBody tag defines the location where content will be placed.
g:layoutHead can be used to allow pages to inject content in the header.
The createLinkTo method makes sure that the webapp directory is automatically added to the URL, so that incorrect use of slashes won’t cause wrong relative urls to default.css.

file: controllers/grailsapp/ProductsController.groovy
class ProductsController {
    def index = {
		[products: s.Service.getProducts()]
	}
}

The method index will be mapped to the URI /products/index, or in short just /products (because it’s the index). Multiple methods (called actions) can be added in this way in the same controller class.
The index method returns a “model”, which is a map that the view uses when rendering. The keys (“products” in this case) within that map translate to variable names accessible by the view.

file: views/products/index.gsp
<g:set var="pageName" value="Product listing" scope="request" />

<html>
	<head>
		<meta name="layout" content="main"></meta>	
		<title>Product listing</title>
	</head>
	<body>
		<table>
			<g:each var="product" in="${products}">
				<tr>
					<td><g:render template="/shared/product" model="[product:product]" /></td>
					<td>
						<g:each var="cat" in="${product.categories}">$${cat.name}, </g:each>
					</td>
				</tr>
			</g:each>
		</table>
	</body>
</html>

The layout meta tag points to the main layout. The html and body tag will be removed from the final output, because those from the main layout will be used.
The g:each tag is used to iterate over the products and categories.
The s:set tag sets a variable so that it can be used in the shared layout.
The g:render tag is used to render products as custom component/elements using Grails templates.

file: views/shared/_product.gsp
<div class="product">
	<img src="${product.name}.jpg" />
	<span class="productname">${product.name}</span>, <span class="price">${product.price}</span>
</div>

This is the custom component (Grails template) to render products.

production settings

grails prod war

Framework: Ruby On Rails

-Ruby 1.9.2-p180
-Rails 3.0.7
-WEBrick 1.3.1 (webserver)
-mongrel 1.2.0.pre2 (webserver)
-JRuby 1.6.2
-Trinidad 1.2.0 (embedded Tomcat for JRuby)

More »

file: config/routes.rb
Railsapp::Application.routes.draw do
	...
	resources :products	
end

This line makes the products controller including all methods (actions) available under URI pattern /products/actionName. Since we are using the index action, which is the default, the URI will default to /products.

file: app/views/layouts/application.html.erb
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="/default.css" media="screen" rel="stylesheet" type="text/css" />
		
		<%= stylesheet_link_tag :all %>
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<!-- Page name needs to be injected -->
					<div><h2><%= @pagename %></h2></div>
					<div class="innertube">
						<%= yield %>

					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

This is the shared template that can be used by multiple pages. The yield method is used to specify the location of where the content of the pages will be placed.

file: app/controller/products_controller.rb
class ProductsController < ApplicationController
	skip_before_filter :verify_authenticity_token
	protect_from_forgery :except => :index
	
	def index
		@products = ProductsHelper::Service.getProducts
	end
end

The skip_before_filter and protect_from_forgery method calls are needed to disable some options that might influence benchmark performance, these options aren’t enabled by default on other frameworks.

The index method is the method that is called at each HTTP request to our page. It creates the @products instance variable on each request, which can than be accessed by the view.

file: app/views/products/index.html.erb
<% @pagename = "Product Listing" %>

<table>
	<% @products.each do |product| %>
	<tr>
		<td class="productname"><%= render "shared/product", :product => product %></td>
		<td class="categories">
			<% product.categories.each do |category| %>
				<%= category.name %>, 
			<% end %>
		</td>
	</tr>
	<% end %>
</table>

As shown above, Rails uses plain Ruby code embedded between <% tags to render Rails code.
The @pagename is used to set the pagename so that it can be used in the shared template/layout.
The render method is used to render product as a custom componenent (called partials in Rails).

file: app/views/shared/_product.html.erb
<div class="product">
	<img src="<%= product.name %>.jpg" />
	<span class="productname"><%= product.name %></span>, <span class="price">$<%= product.price %></span>
</div>

This is a custom component (called a partial in Rails) to render a product.

production settings

Rails Webrick:
-rails server -e production

Rails Mongrel:
-cmdline: gem install mongrel
-cmdline: bundle install
-add line to gem Gemfile: gem “mongrel”, ‘>= 1.2.0.pre2′
-cmdline: rails server -e production

JRuby-Rails Webrick:
-jruby script\rails server

JRuby-Rails Trinidad:
-jruby -S trinidad -e production

Framework: Play

Specs:
-Play 1.2.1

More »

file: views/main.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
    <head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		
        <title>#{get 'title' /}</title>

        <link rel="stylesheet" media="screen" href="@{'/public/default.css'}">
        #{get 'moreStyles' /}
        <link rel="shortcut icon" type="image/png" href="@{'/public/images/favicon.png'}">
        #{get 'moreScripts' /}
    </head>
	
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<!-- Page name needs to be injected -->
					<div><h2>#{get 'pagename' /}</h2></div>
					<div class="innertube">	
						#{doLayout /}
					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

The get tags are used to retrieve variables that are set elsewhere, like in the pages. The doLayout tag specifies the location where content of each page will be shown.

file: controller/Products.java
public class Products extends Controller {
   public static void index() {
		List<Product> products = Service.getProducts();
        render(products);	
    }
}

The index method of the controller that is called on each http request to /products/index.

file: views/products/index.html
#{extends 'main.html' /}
#{set title:'Product listing' /}
#{set pagename: 'Product listing' /}

<table>
	#{list items:products, as:'product'}
		<tr>
			<td>#{product product /}</td>
			<td>
				#{list items:product.categories, as:'category'}
					${category.name}, 
				#{/list}
			</td>
		</tr>
	#{/list}
</table>

The extends tag specifies the shared layout that can also be used by other pages. The set title tag sets the title variable so that it can be used in the shared layout.
The list tag is used to iterate over the products. Instead of a list tag, Groovy code (like Grails) can also be used in the views.
The product tag is a custom component to render a product.

file: views/tags/product.html
<div class="product">
	<img src="${_arg.name}.jpg" />
	<span class="productname">${_arg.name}</span>, <span class="price">$${_arg.price}</span>
</div>

This is the custom component (called a tag) to render a product.

production settings

set to production via the file application.conf:
application.mode=prod
%prod.application.mode=prod

play war -o k:\wars\play –zip

Framework: Lift

Specs:
-Lift 2.3
-Scala: 2.8.1

More »

file: webapp/templates-hidden/default.html
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="default.css" media="screen" rel="stylesheet" type="text/css" />
		
		<!-- Title needs to be injected -->
		<title>Products listing</title>
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<!-- Page name needs to be injected -->
					<div><h2><lift:bind name="pagename">.....</lift:bind></h2></div>
					<div class="innertube" id="content">
						<lift:bind name="content"></lift:bind>
					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

This is the layout that can be shared by multiple pages.
The lift:bind tag is used as a placeholder where data can be placed by other pages or code.

file: bootstrap.liftweb.Boot.scala
class Boot {
	def boot {
		...
		// Build SiteMap
		val entries = List(
			...
			Menu.i("Products listing") / "products",
			...
		)
		...
	}
}

A menu is added to the SiteMap list, this is required for the page to be accessible via an URL. In this case, the name of the menu is “Products listing”, and the URI is /products. It will look for a page at webapp/products.html.

file: webapp/products.html
<lift:surround with="default">
	<lift:bind-at name="pagename">aaaa</lift:bind-at>

	<lift:bind-at name="content">
		<table class="lift:ProductsSnippet.showProducts">
			<tr>
				<td class="productname"></td>
				<td class="categories"></td>
			</tr>
		</table>
	</lift:bind-at>

</lift:surround>

The lift:surround places the body of this tag in the lift:bind tag that has the attribute name=”content” of the template named “default” (webapp/templates-hidden/default.html).
The lift:ProductsSnippet.showProducts value in the class attribute causes Lift to pass the table tag including it’s body to a Scala method in the class “ProductsSnippet” named “showProducts” as an XML object (NodeSeq), or it applies a function that ProductsSnippet.showProducts might return on the table tag. This allows showProducts to perform XML transformations/manipulations at the given table tag.

ProductSnippet.scala
package code
package snippet

import scala.xml.{NodeSeq, Text}
import net.liftweb.util._
import net.liftweb.common._
import java.util.Date
import code.lib._
import Helpers._
import _root_.s._

class ProductsSnippet {

  def showProducts = {
	val products = Service.products
	"tr" #> (in => products.flatMap{ p =>
		(".productname *" #> <div class="product"><img src={p.name + ".jpg"} /><span class="productname">{p.name}</span>, <span class="price">${p.price}</span></div> &
		".categories *" #> p.categories.map(_.name).mkString(", "))(in)
	})  
  }
}

showProducts is a function. The goal of showProducts is to transform the table tag of products.html (including it’s body) to the desired HTML output. In this case, we want the custom product HTML layout (which we call a component) to be placed within the td tag, and we want the categories of each product to be listed in the other td tag.
To do this, we have to iterate over each product, create a custom component/layout for each product, and place it in the td tag. And for each product, we have to iterate over the categories.

showProducts can be difficult to understand if you don’t have experience with functional programming. I’ll try to explain what happens.
Lift takes the table tag of products.html (including it’s body) and calls the showProducts function. showProducts won’t immediately act on the table tag, but it will return a function that will be the result of the function call to the showProducts function. Then Lift passes the table tag as a parameter to the returned function, which will then return the desired HTML output in the form of a NodeSeq object, which will than be placed at the corresponding location in products.html.

(Instead of having showProducts to return a function, it’s also possible to let it receive the HTML (a NodeSeq) immediately, and return the transformed HTML (a NodeSeq). This sounds easier, but the syntax of Lift when returning a function, is shorter and might be easier to write.)

The val keyword declares the “products” variable, which is a List object. “#>” is a method/operator that is added to the String class using Scala’s implicit conversions (via an import statement). The #> method returns a function, but can also be given a function as input parameter.

The first #> method call is used to bind the “tr” tag to HTML (a NodeSeq). The map and flatMap methods are used to translate each product or category to the corresponding HTML output.
The #> method call at “.productname *” binds the new HTML to the body of a HTML element that has “productname” as value of the class attribute.

production settings

set JAVA_OPTS=-Drun.mode=production -Xmx1024m

sbt ~package

Framework: Play-Scala

Specs:
-play-1.2.2RC1
-play-scala-0.9.1

More »

file: app/products.scala (mvc controller)
object Products extends Controller {
    import views.Products._

	def index(n: Int) = html.index(Service.getProducts(n)) 
}

Defines the URI /products/index, and maps it to the index method. The index method takes a parameter ‘n’ that is mapped to the URL querystring parameter ‘n’. Conversions to an integer are applied automatically.
In Play-Scala the view pages/templates are seen as functions (in this case html.index function) that can be passed data (in this case a list of products).

file: app/views/main.scala.html
@(title:String = "", pageName:String = "")(body: => Html)

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
	<head>
		<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
		<link href="default.css" media="screen" rel="stylesheet" type="text/css" />
		
        <title>@title</title>
        <link rel="stylesheet" media="screen" href="@asset("public/stylesheets/main.css")">
        <link rel="shortcut icon" type="image/png" href="@asset("public/images/favicon.png")">
        <script src="@asset("public/javascripts/jquery-1.5.2.min.js")" type="text/javascript"></script>
	</head>
	<body>
		<div id="maincontainer">
			<div id="topsection">
				<div class="innertube"><h1>Company Title .....</h1></div>
			</div>
			<div id="contentwrapper">
				<div id="contentcolumn">
					
					<!-- Page name needs to be injected -->
					<div><h2>@pageName</h2></div>
					<div class="innertube">
					
						<!-- The content that is unique for each page -->
						@body
					</div>
				</div>
			</div>

			<div id="leftcolumn">
				<div class="innertube">
					<h3>Side bar.....</h3>
				</div>
			</div>

			<div id="footer">footer.....</div>
		</div>
	</body>
</html>

In Play-scala pages/templates/tags are seen as functions that can have parameters, so that data can be passed to a page/template/tag.
The first line defines a title, pageName and body parameter. Data can be passed from controllers and other pages/templates/tags.
The “@” sign refers to Scala code.

file: app/views/Products/index.scala.html
@(products: Seq[s.Product])
@import views.tags.html._

@main("Product listing title", "Product Listing page") {
	<table>
		@for(product <- products) {
			<tr>
				<td>@productTag(product)</td>
				<td>
					@for(category <- product.categories.toList) {
						$@category.name,
					}
				</td>
			</tr>
		}
	</table>
}

The first line defines the parameters that can be passed to this page. “main” is a function that refers to the main template. It is passed a title, pageName and the body.

file: app/views/tags/productTag.scala.html
@(product: s.Product)

<div class="product">
	<img src="@(product.name).jpg" />
	<span class="productname">@product.name</span>, <span class="price">$@product.price</span>
</div>

The custom product component (called a tag).

production settings

set to production via the file application.conf:
application.mode=prod
%prod.application.mode=prod

Two production environments were tested:
a) Play-scala running under Tomcat
play war -o k:\wars\play –zip

b) Play-scala running under it’s default builtin webserver called Netty.
JAVA_OPTS=-Xmx1024m
play run

Static file test

The static file test is rendering of plain HTML files served directly by the webserver (Tomcat). So no framework was used in this test.

Test Results

Note: that the more component based frameworks – such as Wicket and Lift – have multiple ways of doing things. Where the simpler MVC push frameworks – such as Rails, JSP, Play, Grails – usually have one way of doing it. Therefore, because there are multiple solutions in Wicket and Lift, there may be faster (and slower) solutions than I have used. However, I did try to use common solutions.

Wicket-1.5rc3 and Wicket-1.5r5trunk use a ListView to iterate over the categories, but Wicketb-1.5rc5trunk uses a RepeatingView.

More »

"u": the number of concurrent users
"p": the number of products to render
"ms": response time in milliseconds
"KB": returned page size in kilobytes.
"%": CPU utilization
"mb": RAM memory usage of the webapp


Grails 1.3.7 - Tomcat
1u, 1000p, 129ms, 205,77 KB, UTF-8
2u, 1000p, 150ms
4u, 1000p, 185ms
8u, 1000p, 428ms
16u, 1000p, 848ms, 155mb, 85%
1u, 0p, 1ms
1u, 1p, 1ms
1u, 10p, 2ms
1u, 100p, 14ms
1u, 500p, 66ms
1u, 1000p, 129ms
1u, 5000p, 629ms

Grails 1.4.0.BUILD-SNAPSHOT (3 June 2011) - Tomcat
1u, 1000p, 107ms, 205,79 KB
2u, 1000p, 115ms
4u, 1000p, 132ms
8u, 1000p, 307ms
16u, 1000p, 622ms, 233mb, 85%
1u, 0p, 1ms
1u, 1p, 1ms
1u, 10p, 2ms
1u, 100p, 12ms
1u, 500p, 55ms
1u, 1000p, 107ms
1u, 5000p, 525ms

Grails 1.4.0.BUILD-SNAPSHOT2 (3 June 2011) - Tomcat
1u, 1000p, 47ms, 205,82 KB
2u, 1000p, 56ms
4u, 1000p, 67ms
8u, 1000p, 149ms
16u, 1000p, 373ms, 197mb, 75%
1u, 0p, 1ms
1u, 1p, 1ms
1u, 10p, 1ms
1u, 100p, 5ms
1u, 500p, 24ms
1u, 1000p, 47ms
1u, 5000p, 229ms

Grails 1.4.0.BUILD-SNAPSHOT (3 June 2011) - Jetty (default builtin)
Same results as Tomcat.


Rails 3.0.7 - Webrick 1.3.1
1u, 1000p, 102ms, 229,08 KB, UTF-8
2u, 1000p, 177ms
4u, 1000p, 254ms
8u, 1000p, 715ms
16u, 1000p, 1438ms, 40mb, 25%
1u, 0p, 15ms
1u, 1p, 15ms
1u, 10p, 15ms
1u, 100p, 16ms
1u, 500p, 52ms
1u, 1000p, 102ms
1u, 5000p, 445ms

JRuby 1.6.2 - Rails 3.0.7 - Webrick 1.3.1
1u, 1000p, 139ms, 229,08 KB
2u, 1000p, 268ms
4u, 1000p, 537ms
8u, 1000p, 1082ms
16u, 1000p, 2252ms, 135mb, 26%
1u, 0p, 2ms
1u, 1p, 2ms
1u, 10p, 4ms
1u, 100p, 16ms
1u, 500p, 71ms
1u, 1000p, 139ms
1u, 5000p, 716ms

Rails 3.0.7 - Ruby 1.9.2-p180 - Mongrel 1.2.0.pre2
1u, 1000p, 110ms, 229,08 KB 
2u, 1000p, 224ms
4u, 1000p, 425ms
8u, 1000p, 583ms
16u, 1000p, 1722ms, 45mb, 25%
1u, 0p, 15ms
1u, 1p, 15ms
1u, 10p, 16ms
1u, 100p, 18ms
1u, 500p, 61ms
1u, 1000p, 110ms
1u, 5000p, 581ms

JRuby 1.6.2 - Rails 3.0.7 - Embedded Tomcat (Trinidad 1.2.0)
1u, 1000p, 142ms, 229,08 KB
2u, 1000p, 144ms
4u, 1000p, 152ms
8u, 1000p, 308ms
16u, 1000p, 621ms, 322mb, 95%
1u, 0p, 2ms
1u, 1p, 2ms
1u, 10p, 3ms
1u, 100p, 16ms
1u, 500p, 73ms
1u, 1000p, 142ms
1u, 5000p, 699ms


JSP 2.1 - JSTL 1.2 - Servlet 2.5 - Tomcat
1u, 1000p, 45ms, 213,58 KB
2u, 1000p, 48ms
4u, 1000p, 54ms
8u, 1000p, 108ms
16u, 1000p, 223ms, 51mb, 85%cpu
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 4ms
1u, 500p, 23ms
1u, 1000p, 45ms
1u, 5000p, 225ms

Static HTML - Tomcat
1u, 1000p, 0s, 213,58 KB
2u, 1000p, 1ms
4u, 1000p, 2ms
8u, 1000p, 5ms
16u, 1000p, 14ms, 72%cpu, 592mb
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 0ms
1u, 500p, 0ms
1u, 1000p, 0ms
1u, 5000p, 4ms

Wicket 1.5-RC3 - Tomcat
1u, 1000p, 251ms, 205,73 KB
2u, 1000p, 506ms
4u, 1000p, 503ms
8u, 1000p, 498ms
16u, 1000p, 499ms, 51mb, 25%cpu
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 2ms
1u, 100p, 19ms
1u, 500p, 119ms
1u, 1000p, 251ms
1u, 5000p, 1885ms

Wicket from trunk (for 1.5-RC5) - Tomcat
1u, 1000p, 251ms, 205,73 KB
2u, 1000p, 263ms
4u, 1000p, 348ms
8u, 1000p, 688ms
16u, 1000p, 1366ms, 158mb, 70%cpu
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 2ms
1u, 100p, 19ms
1u, 500p, 113ms
1u, 1000p, 246ms
1u, 5000p, 1955ms

Wicket from trunk (for 1.5-RC5), RepeatingView version (from Martin G.) - Tomcat
1u, 1000p, 179ms, 220,41 KB
2u, 1000p, 212ms
4u, 1000p, 265ms
8u, 1000p, 544ms
16u, 1000p, 1084ms, 146mb, 70%
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 1ms
1u, 100p, 14ms
1u, 500p, 82ms
1u, 1000p, 179ms
1u, 5000p, 1510m

Wicket from trunk (for 1.5-RC5), Stateful url version (from Martin G.) - Tomcat
NOTE: used with JMeter Cookie manager enabled, otherwise the results will be the same as: Wicket from trunk (for 1.5-RC5), RepeatingView version (from Martin G.) - Tomcat
1u, 1000p, 118ms, 209,67 KB
2u, 1000p, 129ms
4u, 1000p, 141ms
8u, 1000p, 272ms
16u, 1000p, 598ms, 683mb, 90%
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 1ms
1u, 100p, 10ms
1u, 500p, 57ms
1u, 1000p, 118ms
1u, 5000p, 1198ms


Play 1.2.1 - Tomcat
1u, 1000p, 222ms, 197,98 KB, UTF-8
2u, 1000p, 259ms
4u, 1000p, 330ms
8u, 1000p, 676ms
16u, 1000p, 1309ms, 85mb, 85%
1u, 0p, 0ms
1u, 1p, 1ms
1u, 10p, 3ms
1u, 100p, 22ms
1u, 500p, 102ms
1u, 1000p, 222ms
1u, 5000p, 1054ms


Play-Scala 0.9.1 - Play 1.2.2RC1 - Tomcat
1u, 1000p, 26ms, 226,52 KB
2u, 1000p, 30ms
4u, 1000p, 85ms
8u, 1000p, 185ms
16u, 1000p, 406ms, 115mb, 62%
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 3ms
1u, 500p, 14ms
1u, 1000p, 26ms
1u, 5000p, 192ms


Play-Scala 0.9.1 - Play 1.2.2RC1 - Netty
1u, 1000p, 15ms, 226,52 KB
2u, 1000p, 16ms
4u, 1000p, 20ms
8u, 1000p, 42ms
16u, 1000p, 89ms, 572mb, 80%
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 1ms
1u, 500p, 8ms
1u, 1000p, 15ms
1u, 5000p, 79ms

Play-Japid 0.8.3.1 - Play 1.2.1 - Tomcat
1u, 1000p, 4ms, 206,05 KB
2u, 1000p, 6ms
4u, 1000p, 13ms
8u, 1000p, 28ms
16u, 1000p, 63ms, 105mb, CPU too inaccurate to measure
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 0ms
1u, 500p, 2ms
1u, 1000p, 4ms
1u, 5000p, 39ms

Play-Japid 0.8.3.1 - Play 1.2.1 - Netty (builtin)
1u, 1000p, 3ms, 206,03 KB
2u, 1000p, 4ms
4u, 1000p, 5ms
8u, 1000p, 12ms
16u, 1000p, 25ms, 533mb, CPU too inaccurate to measure
1u, 0p, 0ms
1u, 1p, 0ms
1u, 10p, 0ms
1u, 100p, 0ms
1u, 500p, 1ms
1u, 1000p, 3ms
1u, 5000p, 17ms


Lift 2.3 - Scala: 2.8.1 - Tomcat
1u, 1000p, 99ms, 220,74 KB, UTF-8
2u, 1000p, 115ms
4u, 1000p, 138ms
8u, 1000p, 280ms
16u, 1000p, 850ms, starts at 120mb, increases continuously, probably due to a session explosion (each requests creates a new session), 55%
1u, 0p, 10ms
1u, 1p, 10ms
1u, 10p, 10ms
1u, 100p, 17ms
1u, 500p, 52ms
1u, 1000p, 99ms
1u, 5000p, 633ms

Tapestry 5.2.5
1u, 1000p, 20ms, 171,41 KB
2u, 1000p, 24ms
4u, 1000p, 30ms
8u, 1000p, 65ms
16u, 1000p, 192ms, 147mb, 65%
1u, 0p, 1ms
1u, 1p, 1ms
1u, 10p, 1ms
1u, 100p, 3ms
1u, 500p, 10ms
1u, 1000p, 20ms
1u, 5000p, 100ms

Context Framework 0.8.1 / Benchmark 1.0-SNAPSHOT
1u, 1000p, 60ms, 186,88 KB
2u, 1000p, 68ms
4u, 1000p, 133ms
8u, 1000p, 367ms
16u, 1000p, 768ms, 144mb, 41%
1u, 0p, 1ms
1u, 1p, 1ms
1u, 10p, 2ms
1u, 100p, 7ms
1u, 500p, 31ms
1u, 1000p, 60ms
1u, 5000p, 325ms

Grails 1.3.7 - Freemarker 0.3 - Freemarker plugin 0.6.0
1u, 1000p, 22ms, 205,8 KB
2u, 1000p, 32ms
4u, 1000p, 43ms
8u, 1000p, 99ms
16u, 1000p, 299ms, 200mb, 60%
1u, 0p, 2ms
1u, 1p, 2ms
1u, 10p, 2ms
1u, 100p, 4ms
1u, 500p, 12ms
1u, 1000p, 22ms
1u, 5000p, 112ms



Test Remarks

CPU Utilization is inaccurate
More »

JMeter impacts performance results because JMeter needs to collect data and calculate statistics. This is more prominent the faster the response times are, because then more data will be generated. CPU utilization of JMeter sometimes reaches 50%. In these cases, my CPU utilization of the associated framework will be very inaccurate.
Rails
More »

During testing, I noticed some high peeks in response time. This might be due to Garbage collection of Ruby. I measured the response time without including these extreme pauses.
This is more difficult to test when performing a 2 concurrent user test. In this case, Rails was showing 2 groups of response times, where the second group was about half the response time of the first group. The groups seems to interleaf in series.

Commenters have pointed out that Rails should be tested under other webservers than Webrick.

Lift
More »

cpu utilization about 53%, where other webapp have about 85%
Lift required more initial benchmarking time before reaching stable response times, otherwise the average responsetime will drop slowly.
At 16u, the response times fluctuate a lot. I have seen long running averages at 670ms, but also at 950ms, then slowly creeping up, then slowly down etc. This might be due to barbage collection and the effect of starting at 0 sessions or measuring when a higher amount of sessions are present (even though I’m using JMeter without cookies/sessions enables). Because of this, I have set it to 850ms.
A LOT of samples are needed.
Wicket
More »

During testing at 8u and 16u, wicket gave multiple warnings/exceptions:
WARN – PageAccessSynchronizer – “http-apr-8080″-exec-16 failed to acquire lock to page 0, attempted for 1 minute out of allowed 1 minute
Caused by: org.apache.wicket.page.CouldNotLockPageException: Could not lock page 0. Attempt lasted 1 minute

A post commenter (@12) pointed that this is due to a bug that is fixed in the latest trunk version of Wicket. This trunk is tested under “Wicket-trunk” in this post.

Tapestry
More »

Tapestry iterates over the categories using plain Java API (StringBuilder) instead of Tapestry. Because according to the writer of the test code (Kaosko) “the general guideline for minimizing logic in the template”. I expect that the performance of Tapestry will be lower when Tapestry solutions are used to iterate over the categories.
The benchmark test case used to test all frameworks, should be changed to use custom components for categories instead of using plain Strings to render categories.

Also note that even though I try the HTML output to be roughly the same in size, the output size of Tapestry was lower then the rest: 171,41 KB. Tapestry seems to remove excessive whitespaces.

Effects of page size by unneeded whitespaces
More »

As an example, Play-scala-netty is used to show the effects, but the same probably applies to other frameworks.
During testing, I accidentaly forgot to check the page sizes of Play-scala, and tested Play-scala. Later I saw that the page size was: 373,04 KB (for 1000 products). To reduce this size to approximately the same size in which I tested other frameworks, I removed some unneeded whitespaces from products.scala.html, by changing:

@for(category <- product.categories.toList) {
	$@category.name,
}

to

@for(category <- product.categories.toList) { $@category.name, }

The page size then was reduced to: 226,52 KB.

Here the differences in response time for:
users, products, 226.52kb vs 373,04kb
1u, 1000p, 15ms vs 21ms
2u, 1000p, 16ms vs 23ms
4u, 1000p, 20ms vs 30ms
8u, 1000p, 42ms vs 64ms
16u, 1000p, 89ms vs 137ms
1u, 0p, 0ms vs 0ms
1u, 1p, 0ms vs 0ms
1u, 10p, 0ms vs 1ms
1u, 100p, 1ms vs 2ms
1u, 500p, 8ms vs 10ms
1u, 1000p, 15ms vs 21ms
1u, 5000p, 79ms vs 111ms

(With Tomcat gzip compression enabled, the page size is reduced to a page size of about 9kb.)

JMeter

JMeter impacts the test results. This might be due to JMeter using RAM or CPU. I should retest the tests with JMeters Distribution Graph disabled.

Package sizes

grails.war, 22.088kb
jsp.war, 375kb
lift.war, 14.111kb
play.war, 29.124kb
playscala.war, 53.119kb
wicket.war, 2.200kb

Discussion

What suprised me is that even though JSP/JSTL, Rails, Grails and Play use about the same MVC model, the differences in performance are big. Simply switching from template system in Play framework from the default to Japid, has a huge impact (in this test 27 times faster).

Most posts about JRuby on Rails tell that it is faster than Ruby on Rails, however in this test, Ruby on Rails was clearly faster in both the rendering of products and concurrent-user test.
Also, most posts about Rails tell that Webrick is slower than Mongrel, but in this test, Webrick was a bit faster.
JRuby on Rails running on Trinidad does perform faster than other Rails configurations when tested with concurrent users.

Play-Scala using Netty NIO server is faster than Tomcat webserver. I wouldn’t expect the differences to be that big. It’s unclear whether these fast results are caused by Play-Scala being optimized for Netty, or that Netty is simply faster than Tomcat (and thus that other frameworks could also have faster results under Netty).
Note that the memory usage of Play-Scala under Netty is a lot higher than under Tomcat.

The section “Effects of page size by unneeded whitespaces” shows that unneeded whitespaces impact the response time. Frameworks can probably be optimized by stripping unneeded whitespaces from the templates during boot, even when HTTP gzip compression is enabled. (During the test, compression was disabled in Tomcat and Netty.)

Questions

-Does anybody know why the static file test under Tomcat is using so much memory?

Blog post updates

-30 May 2011: Added Play-scala (under Tomcat webserver) and Play-scala-netty (under Play’s builtin Netty webserver). Added the effects of unneeded whitespaces. Added Wicket-trunk, which is the latest Wicket snaphot from trunk (for 1.5-RC5), and info about a performance bug in Wicket.
-31 May 2011: Added Play-Japid and JRails.
-3 June 2011: Added Grails trunk.
-13 June 2011: Added Tapestry, Context-framework and Grails-1.3.7-freemarker

Project files

The files (code, project, html) of this post can be found here: https://github.com/jtdev/blogpost_files

  123 Responses to “Rails, Wicket, Grails, Play, Tapestry, Lift, JSP, Context”

  1. Hi,

    You have used the old groovy template system for the Play framework. Can you please test it with the Play’s new native template system. It’s only available for scala now but will be available for java in the future, too.

    Serdar

  2. Few comments about rails:
    – webrick is slow, for ruby 1.9 use thin (or unicorn, passerger in real production)
    – rails (ruby) is not concurent, su you were runing only one process, this mean one user at a time (thin supports running few servers)
    – not sure what was your products model, but you definetely should have done this in controller: Product.includes(:categories).all – this eliminates n+1 query.

  3. @sirmak
    I have justed tested your suggestion using the Scala version of Play by building a new Play-scala project. The view code is the same, but controller code differs in syntax and API. (I’m using the template as documented at the Play-scala website. I’m not sure if this is the new native template system.)

    Test parameters:
    -16 concurrent users
    -rendering 1000 products

    The previous results under Play 1.2.1:
    -Average response time: 1309ms
    -JVM usage: ~85mb
    -CPU utilization: ~85%
    -war size: 29.124kb

    The results for Play 1.2.1 scala-0.9:
    -Average response time: 1701ms (mean), deviation: 128
    -JVM usage: ~102mb
    -CPU utilization: ~85%
    -war size: 45,9 MB

  4. @admin

    thank you very much for your attention, the released scala version was a bit old and the new version is at https://github.com/playframework/play-scala and not released for now.

    the templating system with play scala 0.9 was old groovy template views, too. The new version on the github uses a new native scala template system (no moore groovy) with asp.net razor like syntax. I expect far better result with this new system, I appreciate you if you can can test this new development.

    my best
    Serdar

  5. @Julius,

    1,2)
    I have just tried to install Unicorn and Thin, but had some installation problems (maybe not compatible with Windows?).
    I read that Mongrel gives better performance than Webrick, and that Mongrel is often used in production. So I have just tried Mongrel.

    -cmdline: gem install mongrel
    -cmdline: bundle install
    -add line to gem Gemfile: gem “mongrel”, ‘>= 1.2.0.pre2′
    -cmdline: rails server -e production

    The response times are similar to Webrick. Maybe Mongrel is a lot faster, but in this test case (16 users, 1000 products) other factors might be the bottleneck?

    3)
    I’m using the following code to get the products and category objects:
    https://github.com/jtdev/blogpost_files/blob/master/railsapp/app/helpers/products_helper.rb
    Everything is in-memory, and the object network is build up only once at the first request, so the n+1 query problem shouldn’t exist.

  6. @Sirmak
    I’m getting errors installing the unreleased Play-scala from git following the steps from http://scala.playframework.org/code . I think the build system is developed for Linux/Unix based systems, which make it hard to install it from Git.

  7. @admin

    thank you very much again Admin, you’re really appreciated. I wish I can help, but I’m only a happy Play user, I think a Play developer or a better Play user can help if they see this great article.

    Best Regards

  8. Ok, so I’ve run some tests (with Apache Benchmark) on my 64bit linux system (AMD Phenom@2.8GHZ). 1000 requests with 16 users concurrency took 120s. 16 users with 16 concurrency just about 1.9s and 1000 requests with one user at a time – 113s. RAM usage was avg 56mb for 16conc and avg 48mb for 1conc.
    Also please note that ruby is not good at concurrency – it cannot use more then one processor.

    But JRuby can – because it’s JVM, first run on JRuby was about 2 times slower than 4th (i think it should be similar to other JVM based frameworks). Running 1000 requests with concurrency 16 took about 119s. But memory usage was insane – 300mb.

    But overall it’s nice to see that rails/ruby does not get it’s ass kicked by Java frameworks.

  9. Thanks for sharing these useful stats.

  10. Excellent article

    I hope you can mantain it, for future comparisons. I’m also a happy play user, and the current groovy templating system is one of the most discussed topics, I’m also very anxious to test the next groovy-less templating system, so far now it’s only available for scala…

    saludos

    sas

  11. @admin

    I’ve excellent news, thanks to Play developers, Play 1.2.2RC1 and Play Scala 0.9.1 released, this new release include the new native scala templating system. can you please make your benchmark with this new release http://scala.playframework.org/

    And please use play’s own http server (netty) for optimal performance, no need to use an application server like tomcat (and no need to convert to war).

    Best Regards,
    Serdar

  12. Some notes about Wicket:
    a) 1.5-RC3 and later versions all had a performance problem which was only recently fixed in https://issues.apache.org/jira/browse/WICKET-3740. Please build from trunk and try again.

    b) you do not use detachable models – this means that wicket may have to serialize the entire list of products into the session. Serialization is slow and this may impact your test results.

    Please fix the two problems above, and try again. I would be interested at seeing the difference.

    Cheers.

  13. Thank you, that was an useful comparison. My company’s web application with hundreds of included jsp templates (jstl and el) it’s taking between 10ms and maybe 40ms in just rendering the html. Some fast tests with glassfish and resin showed no improvement, and my guess is that the only way to speed thing up is caching fragments.

  14. JSP has been around the longest, so it doesn’t surprise me too much it came out on top. I’d be curious to see the performance you can squeeze out of the Tomcat apps if you added the Apache Portable Runtime (APR) library to Tomcat. This gives Tomcat native JNI/native access to the OS’s core features, making it even faster.

    Wish we could have see a JSF2.0 app in the mix…

    Glad to see Rails came out near the top!

  15. This is not surprising.

    So many of these frameworks just add a ridiculous amount of bloat.

    If you take the JSP example and just use scriptlets instead of JSTL, you’ll get even better performance.

  16. Especially under Windows the MRI Ruby implementation is slow. For a fair comparison I would recommend JRuby 1.6.2 + Trinidad (Tomcat wrapper), so that Ruby performance and Concurrency handling is done the way it should.

    Using 1.9.2 and Webrick on Windows is especially bad. The suggested Unicorn, Passenger, … only run under UNIX Systems as far as I know.

    http://www.jruby.org/
    https://github.com/trinidad/trinidad

    Updating your app should only be a matter of 5 min.

  17. @12
    b) The Wicket page is stateless (if I’m correct) (https://cwiki.apache.org/WICKET/stateless-pages.html), because I’m not using stateful components. Also, no sessions are created, and the Product class isn’t serializable, so serialization shouldn’t/can’t occur. For safety, my latest test in Wicket uses a LoadableDetachableModel. Some quick tests showed no performance difference using the same version of Wicket (1.5-RC3).

    a) Wicket from trunk didn’t output the PageAccessSynchronizer warnings anymore. However, I’m now getting mixed results:

    Previous performance (as seen in the first post results) (with the PageAccessSynchronizer warnings):

    1u, 1000p, 251ms, 205,73 KB
    2u, 1000p, 506ms
    4u, 1000p, 503ms
    8u, 1000p, 498ms
    16u, 1000p, 499ms, 51mb, 25%cpu
    

    (The horizontal curve is very strange behaviour.)

    Performance from trunk:

    1u, 1000p, 217ms, 205,73 KB
    2u, 1000p, 256ms
    4u, 1000p, 336ms
    8u, 1000p, 698ms, 135mb, 68%cpu
    16u, 1000p, 1362ms
    

    The trunk version is following a more expected behaviour/curve of the response time. However, at 8 and 16 users, the response time is higher than 1.5-RC3. Also, CPU utilization is higher.

    The performance at 1 user, and 500 and 1000 products is about 13% faster using the trunk.

  18. I have just updated the blog post with Play-scala and Wicket-trunk.

  19. @admin

    thank you very much, great article, great work.

    Serdar

  20. The memory problem with play-scala netty is that the server is started with -Xms512m. Try play run –%prod -Xms50m.

  21. Can we also see the play version running on netty (the default in play)? The test should run in prod mode (–%prod). That will be nice so we can see the role that Netty plays in the performance and also the part the groovy templating engine plays.

    Thanks again

  22. It would also be nice to see Play (java) using default Netty, but using the Japid Java templating engine. Japid is pretty easy to use, but I would be willing to setup a package for you to run on you system to maintain the same environment for the test.

  23. @Nicolas
    Play was set to prod mode via application.conf.

    In the first post, I have tested Play-1.2.1-tomcat (Groovy templates). Then on request of a commenter, I tested Play-1.2.1-Scala-0.9-tomcat (Groovy templates), but only in a short test (see comment #3). Then I tested Play-1.2.2RC1-Scala-0.9.1-tomcat (Scala templates) and Play-1.2.2RC1-Scala-0.9.1-netty (Scala templates).

    Play-1.2.1-Scala-0.9-tomcat vs Play-1.2.2RC1-Scala-0.9.1-tomcat shows the performance difference of the template systems; it shows that the Scala templates are a lot faster than the Groovy templates (assuming that Play-1.2.2RC1 by itself didn’t change the performance significantly compared to Play-1.2.1, and assuming that Play-Scala-0.9.1 by itself excluding the Scala templates didn’t change the performance significantly compared to Play-Scala-0.9.).

    And Play-1.2.2RC1-Scala-0.9.1-tomcat vs Play-1.2.2RC1-Scala-0.9.1-netty shows that Play runs (by default) faster under Netty than under Tomcat.

    @luizalvino
    I’m unable to see any difference using any -XmsXXXXm or -XmxXXXXm setting, like the settings are ignored. For example, starting Play using:
    play run -Xms50m -Xmx50m
    should start and limit memory usage to at most 50mb. (If I’m correctly understanding Java’s Xms and Xmx.)
    I have tested these settings using both the command line and via application.conf (jvm.memory=-Xms50m -Xmx50m).

  24. Thanks a lot admin for the updates, the play-japid experiment would be very interesting too… I’m still pretty much surprised about play-scala performance agains jsp…

  25. Can we have a test for grails 1.4 trunk (http://github.com/grails/grails-core). The best way to test it is to create a new app and copy the src and controller artefact (and if you use a custom view too). Then a simple prod run-war should display interesting results.
    There are several improvements like static invokations in GSP, static injections in controllers, static resources cache… on my test with an empty page for instance :
    1200req/s Grails 1.3.7
    2250req/s Grails 1.4 Snapshot.

    As you said, using caching ( like the super easy/conventional grails plugins Spring Cache + Resources ), pimp performances is really simple.

  26. I have just done some test with Play-1.2.1-japid-0.8.3.1 (Thanks to Bing/Branaway https://github.com/jtdev/blogpost_files/pull/1). Results using Tomcat:
    16u, 1000p, 63ms
    1u, 5000p, 39ms
    Under Netty, it’s even faster.
    I’m going to update the blog post with it.

    I’m also going to test JRuby on Rails and Grails 1.4 trunk.

  27. Hi,

    Thanks for the tests and the posted results!

    I’m investigating why Wicket is slow and I see that the app makes 5000 requests to non-existing images.
    I see such images only in Wicket and Lift apps.
    I’ll send you Pull Request for all mine findings.

  28. The comparison is somewhat unfair, since Wicket is the only component-based framework (that have to build a component tree, evaluate events, etc.).

    The others are action-based frameworks, which are, basically, syntax sugar to println().

  29. Actually using stateful Wicket page will improve the result because the component tree will be generated just once and then reused for all following requests and just rendering will be measured as with the other frameworks.

  30. Great comparison. Any chance you could add Tapestry 5 to the results? If you don’t feel comfortable with writing the test app in Tapestry, I can do it for you. Feel free to ping me.

  31. It would be interesting to see Tapestry in this comparison. I could help you create the example. What do you think?

  32. I was also surprised that static file serving is way faster than all the frameworks

    Why?

  33. Updated the blog post with Play-Japid, JRails and Rails under Mongrel and Webrick.

    @ Kalle and/or Derkoe
    About Tapestry 5, I don’t have experience with it. Would be great if you can provide a Tapestry version at GitHub?

    @Tetsuo
    Imo everything has to be seen in context, including this test. This test doesn’t tell which framework is the best in general, or which is the fastest in general. It only tests frameworks under the given test conditions. And that condition could be what some developers want to see tests for.
    But I understand your point. Personally I also prefer component based frameworks, but I question the assumption that component based frameworks are always slower than MVC push frameworks. Btw, Lift can also be seen as a component based framework, although Wicket is more component-based.

    @martin-g
    I’m going to test your Wicket samples tomorrow.

  34. @32, Dave
    “I was also surprised that static file serving is way faster than all the frameworks” “why?”
    I didn’t expect the differences to be that big. But now, after having seen the results of Play-Japid, I should change that statement.

  35. Oh my god! Play+Japid on Netty-based NIO Server is that fast, even faster than pure JSP and only behind serving static files.

    I think, there is on the world only another one combo (Web/Application-Server + Simple Web Framework), that would outperformt even Play+Japid+Netty … and it would be G-WAN with its C Servlets. A awesome software piece created by Piere.

  36. @Tetsuo, yes it might be unfair, but no one on earth would care if your site is based on action or component except you, if I’m not wrong. For an user, all it matters is if pretty fast and for a developer if easy to develop. The above ones are pretty good at these :-)

  37. Rails, by default, does not run in threaded mode. However, when using JRuby you get a substantial performance boost when running in threaded mode. I found this out the hard way. You have to set min and max runtimes to 1 in trinidad.yml, and put config.threadsafe! in your environments/production.rb. Oh, and only test performance in production mode.

  38. @Martin-g
    I have just tested your wicket version (with has an RepeatingView component to iterate over the categories instead of a ListView, and some other changes). However, I did add the img html tag, because the other frameworks were also tested using that tag. (The missing images are no problem because the JMeter benchmark isn’t loading trying to download any image.)

    Results:

    Wicket from trunk (for 1.5-RC5), RepeatingView version (from Martin G.)
    1u, 1000p, 179ms, 220,41 KB
    2u, 1000p, 212ms
    4u, 1000p, 265ms
    8u, 1000p, 544ms
    16u, 1000p, 1084ms, 146mb, 70%
    1u, 0p, 0ms
    1u, 1p, 0ms
    1u, 10p, 1ms
    1u, 100p, 14ms
    1u, 500p, 82ms
    1u, 1000p, 179ms
    1u, 5000p, 1510m
  39. @25, Stephane
    I’m unable to test Grails 1.4.0.M1. Both the trunk and the zip give the error when running the app with “grails run-app”:
    “NoClassDefFoundError: org/hibernate/annotations/common/reflection/ReflectionManager”

  40. @37, Mark Thomas
    All frameworks were run in production mode, including Rails.

    Rails-3.0.7-Mongrel:
    Enabling config.threadsafe! does seem to give faster results for Rails-3.0.7-Mongrel, but only marginal. According to http://m.onkey.org/thread-safety-for-your-rails there are many factors that should be watched out for when using config.threadsafe.

    JRuby-on-Rails-Trinidad:
    Enabling config.threadsafe! in combination with min_runtimes and max_runtimes set to 1, resulted in the same or a bit slower performance.

  41. You should add Spring in this test, not just JSP-Tomcat.

  42. Thank you for benchmark!
    Play-Japid-Netty combination is definitely worth to look at!
    And I support tunggad – putting G-WAN to the list would give global picture to analyze on.

  43. @wonhee

    I think, Spring MVC would be somewhere abit slower than pure JSP+Tomcat. Ok, it also depends on which view technology (JSP, Freemaker, StringTemplate) we use, but for JSP it would be abit slower than pure JSP.

  44. @admin
    mmmh it works for me, did you make a grails upgrade or a new grails app ?
    Here my steps from trunk :
    -git clone git://github.com/grails/grails-core.git
    -cd grails-core
    -./gradlew assemble
    – cd ..
    – cp grails-core/build/distributions/grails-1.4.0.BUILD-SNAPSHOT.zip .
    – unzip grails-1.4.0.BUILD-SNAPSHOT
    – export GRAILS_HOME=[…]/grails-1.4.0.BUILD-SNAPSHOT
    – rm -rf $USER_HOME/.ivy2/cache/org.grails* //safe step in case of weird errors
    – make sure GRAILS_HOME and PATH:GRAILS_HOME/bin use the unzipped dir
    – grails create-app grailsapp14
    – copy old views, Service sources and controller only (do not copy configuration)
    – grails war

  45. However we are on track to bring more perfs for Grails 1.4 GSP (specially the use of render:template which is less optimized than a tag).

    Another new stuff for Grails controllers :
    -you can declare them as singleton on class level or application config :
    static scope=”singleton”
    -you can use a public method instead of closure for actions (it helps singleton scope since a closure is not thread safe).

  46. @35, Tunggad @42, Steve
    If I’m correct, G-Wan is a webserver that can call Ansi C scripts, and it comes with a library of low-level http operations.

    I think G-Wan isn’t suited for the test case, because the case assumes a framework that has the following features:
    -Object oriented, but this shouldn’t be a big issue, because OO shouldn’t impact performance much compared to procedural.
    -Separation of a page into templates. If I’m correct, you have to do this completely yourself in G-Wan, because these features aren’t provided.
    -Able to make custom components. Same as for templates.
    -Testable under Windows
    -Able to run the webframework under multiple webservers.

    @44, Stephane Maldini
    Got the same problem using your steps, but it hinted me on this, which solved the problem:
    rm -rf $USER_HOME/.ivy2/cache/org.grails*
    should be
    rm -rf $USER_HOME/.ivy2/cache/*

    @45, Stephane Maldini
    Grails 1.4 does seem to be a bit faster. I’ll post the results.
    A singleton controller and public method didn’t show a noticeable performance improvement.

  47. @Martin G.

    Here the results of Wicket, including your last Wicket samples:
    [hide-this-part]

    Wicket 1.5-RC3 - Tomcat
    1u, 1000p, 251ms, 205,73 KB
    2u, 1000p, 506ms
    4u, 1000p, 503ms
    8u, 1000p, 498ms
    16u, 1000p, 499ms, 51mb, 25%cpu
    1u, 0p, 0ms
    1u, 1p, 0ms
    1u, 10p, 2ms
    1u, 100p, 19ms
    1u, 500p, 119ms
    1u, 1000p, 251ms
    1u, 5000p, 1885ms
    
    Wicket from trunk (for 1.5-RC5) - Tomcat
    1u, 1000p, 251ms, 205,73 KB
    2u, 1000p, 263ms
    4u, 1000p, 348ms
    8u, 1000p, 688ms
    16u, 1000p, 1366ms, 158mb, 70%cpu
    1u, 0p, 0ms
    1u, 1p, 0ms
    1u, 10p, 2ms
    1u, 100p, 19ms
    1u, 500p, 113ms
    1u, 1000p, 246ms
    1u, 5000p, 1955ms
    
    Wicket from trunk (for 1.5-RC5), RepeatingView version (from Martin G.) - Tomcat
    1u, 1000p, 179ms, 220,41 KB
    2u, 1000p, 212ms
    4u, 1000p, 265ms
    8u, 1000p, 544ms
    16u, 1000p, 1084ms, 146mb, 70%
    1u, 0p, 0ms
    1u, 1p, 0ms
    1u, 10p, 1ms
    1u, 100p, 14ms
    1u, 500p, 82ms
    1u, 1000p, 179ms
    1u, 5000p, 1510m
    
    Wicket from trunk (for 1.5-RC5), Stateful url version (from Martin G.) - Tomcat
    NOTE: used with JMeter Cookie manager enabled, otherwise the results will be the same as: Wicket from trunk (for 1.5-RC5), RepeatingView version (from Martin G.) - Tomcat
    1u, 1000p, 118ms, 209,67 KB
    2u, 1000p, 129ms
    4u, 1000p, 141ms
    8u, 1000p, 272ms
    16u, 1000p, 598ms, 683mb, 90%
    1u, 0p, 0ms
    1u, 1p, 0ms
    1u, 10p, 1ms
    1u, 100p, 10ms
    1u, 500p, 57ms
    1u, 1000p, 118ms
    1u, 5000p, 1198ms
    

    [/hide-this-part]
    But I do not see the last sample as a regular sample for the test case, because it depends on sessions, and the improved results will probably only show up when same callback is made to the page, like a form posts. When cookies are disabled (which most search engines probably have), the results will be the same as the previous results except now there will be a session explosion (a new session for each http request).

  48. @Admin
    Which trunk version did you use ? We have doubled perf today for pages which use :)
    http://jira.grails.org/browse/GRAILS-7582

    Thanks to your benchmark !

  49. @48, Stephane
    The trunk that I used today most have been just before you guys doubled the performance, because now I’m getting this :)

    Grails 1.4.0.BUILD-SNAPSHOT (3 June 2011) - Tomcat
    1u, 1000p, 107ms, 205,79 KB
    2u, 1000p, 115ms
    4u, 1000p, 132ms
    8u, 1000p, 307ms
    16u, 1000p, 622ms, 233mb, 85%
    1u, 0p, 1ms
    1u, 1p, 1ms
    1u, 10p, 2ms
    1u, 100p, 12ms
    1u, 500p, 55ms
    1u, 1000p, 107ms
    1u, 5000p, 525ms
    
    vs
    
    Grails 1.4.0.BUILD-SNAPSHOT2 (3 June 2011) - Tomcat
    1u, 1000p, 47ms, 205,82 KB
    2u, 1000p, 56ms
    4u, 1000p, 67ms
    8u, 1000p, 149ms
    16u, 1000p, 373ms, 197mb, 75%
    1u, 0p, 1ms
    1u, 1p, 1ms
    1u, 10p, 1ms
    1u, 100p, 5ms
    1u, 500p, 24ms
    1u, 1000p, 47ms
    1u, 5000p, 229ms
    
  50. @Admin : awesome ! it places Grails on par with JSP which is IMO pretty cool for a dynamic framework and rich features like Grails.
    We’re still tracking better optimizations, thanks for your post again, it helped us a lot !

  51. Updated the performance graphs and GitHub.

    @Stephane
    I’m glad that the blog post helps the communities improving the frameworks :)

  52. I am pretty interested in how Play-1.2.1+Netty performs. Can you please add that?

  53. @admin
    Thank your very much for the work!

    Its really nice to see that Grails improves itself over each version, that is advantage of hanving a big company backing a opensource project with paid developers.

    Could you pls bench Grails 1.4 again but with Groovy++ along? There is a groovypp-all-xxx.jar, which is compatible with the groovy-all.jar of Grails. You can download it fron . From the WiKi page you will also find the guide how to integrate it with Grails.

    –> G-WAN: Support for web programming of G-WAN is now very low-level, rudimentary, its just like Servlet of Java World. Therefor its name is C-Servlet too. But it provides the base to build any framework upon it, in C Servlets (ANSI C Scripts) one can call/use any C libs. But for now its abit unfair to bench something low-level like that with other high-level frameworks, its true.

    Thank you very much!

  54. thanks a lot for adding play-japid to the test… really awesome result, btw

  55. Nice benchmark :D

    I would like to introduce test results from a component based framework called Context.


    1u, 1000p, 62ms
    2u, 1000p, 119ms
    4u, 1000p, 219ms
    8u, 1000p, 405ms
    16u, 1000p, 745ms

    1u, 0p, 3ms
    1u, 1p, 3ms
    1u, 10p, 3ms
    1u, 100p, 9ms
    1u, 500p, 33ms
    1u, 1000p, 62ms
    1u, 5000p, 445ms

    I made the tests with Apache Benchmark, so I do not know how it compares with JMeter.

    The test can be duplicated by going here:

    https://github.com/contextfw/contextfw/tree/master/devel/benchmark

    I did some framework optimizations based on this benchmark and was able to gain some more speed. :-)

  56. @tungadd, Grails 1.4 + g++ changes nothing for this bench, the optimization we bring are already transforming most of GSP dynamic dispatch into static dispatch.
    FYI : I gain like 1-5% for @Typed in the generated GSP and controllers.

    Btw we are working on some others optimization before 1.4 final, more on the servlet and binding part which now take most of time here.

  57. @Admin

    Congrats, you have just created a reference benchmark..

  58. […] a second server option for the Rails test, trying out the new Scala-based templates for Play, and based on feedback from Stéphane Maldini running the tests using the latest 1.4 Grails code. Stéphane and Lari Hotari had already been […]

  59. Congratulations Groovy & Grails Team!

    Go Grails Go!

  60. Thank you for the benchmark. It’s surprising to see a framework using static languages like Play-Scala is a bit slower than Grails 1.4

  61. @Zach

    – GSP are mainly statically dispatched now
    – Spring MVC (under the hood) is very efficien,
    – Groovy 1.8 got nice updates in the performances area.

    I did some new optimizations I’m testing them right now but the bench already gives me 30% better results.

  62. How does Wicket 1.4 (the production version) fair?

  63. I am almost giving up to grails due to slow startup and thinking that it might be slow in performance. However, reading at your benchmark, I become aware of new optimizations in grails work great. Thank you for the grails team who has been looking into this blog seriously and watching the progress to improve the framework.

    @Stephane
    I look forward to new improvements in grails.

  64. […] of tweets and retweets this week about a rendering performance comparison between Rails, Wicket, Grails, Play, Lift, and JSP. The blog author updated the post and benchmarks […]

  65. Question: Which operational system do you run this tests?

  66. Hi.

    If you are using JMeter, I think it is probably a good idea to change the “Patterns to Test” in “Response Assertion”: You must add “footer…..” or replace “Company” by “footer…..”, to make sure that the complete response was generated.

    Just my 2 cents.

    Best regards,

    Daniel.

  67. Do you intend to update the contents and graphs to reflect the changes coming from all the comments ?

  68. Kalle did the Tapestry implementation: https://github.com/jtdev/blogpost_files/pull/4

    Could you give it a try?

  69. I only wish you’d have tested JSF for comparison purposes. This is the de-facto Java standard for doing web applications. I personally hate jsf (http://ihatejsf.com) and would like to know more about the actual performance of it.

  70. Grails 1.3.7 with FreeMarker is a viable option for this particular case. The performance is likely to coincide with Grails 1.4 and JSP:

    http://techdm.com/grails/?p=445&lang=en

  71. Can someone exlain how play java is slower than grails 1.4? Seems counter intuitive, no?

  72. Already said – In this test case, most of generated source is static (except product binding). Moreover, Grails bring buffer optimizations + Spring MVC which is relatively fast.

  73. @numan

    It is not java vs groovy speed comparison here. It is groovy vs groovy.

    play-1.2.1 test uses template engine written in groovy (see here: http://www.playframework.org/documentation/1.2.1/templates)

    groovy-1.4 uses own template engine (gsp) written in groovy.

    So, if most of test workload concentrates on rendering part of frameworks, we can expect that results of two groovy based template engines will be comparable.

  74. The stats of Tapestry 5.2.5:

    1u, 1000p, 20ms, 171,41 KB
    2u, 1000p, 24ms
    4u, 1000p, 30ms
    8u, 1000p, 65ms
    16u, 1000p, 192ms, 147mb, 65%
    1u, 0p, 1ms
    1u, 1p, 1ms
    1u, 10p, 1ms
    1u, 100p, 3ms
    1u, 500p, 10ms
    1u, 1000p, 20ms
    1u, 5000p, 100ms
  75. The stats for Context Framework 0.8.1 / Benchmark 1.0-SNAPSHOT:

    1u, 1000p, 60ms, 186,88 KB
    2u, 1000p, 68ms
    4u, 1000p, 133ms
    8u, 1000p, 367ms
    16u, 1000p, 768ms, 144mb, 41%
    1u, 0p, 1ms
    1u, 1p, 1ms
    1u, 10p, 2ms
    1u, 100p, 7ms
    1u, 500p, 31ms
    1u, 1000p, 60ms
    1u, 5000p, 325ms
  76. The stats for Grails 1.3.7 – Freemarker 0.3 – Freemarker plugin 0.6.0:

    1u, 1000p, 22ms, 205,8 KB
    2u, 1000p, 32ms
    4u, 1000p, 43ms
    8u, 1000p, 99ms
    16u, 1000p, 299ms, 200mb, 60%
    1u, 0p, 2ms
    1u, 1p, 2ms
    1u, 10p, 2ms
    1u, 100p, 4ms
    1u, 500p, 12ms
    1u, 1000p, 22ms
    1u, 5000p, 112ms
  77. updated the graphs with Tapestry, Grails-fm and Context framework

  78. @65, Wanderson Santos
    See the section Methods > Test system, for the system specs.

    @66, Daniel
    After I saw the strange curve of Wicket in the concurrent user test, I changed the “Response Assertion” to “footer”, which I also used for the other frameworks. (But I didn’t add it to GitHub, but now it should show the newer JMeter file.)

    @70, Padcom
    If you can provide a JSF 2 sample, I can test it.

  79. FYI there are tiny optimisations for Grails 1.4 in the latest builds. Feel free to test them but it’s not very important. Atm the bench is very complete and you should have already spent too much time !

    Thx you again :)

  80. Thanks a lot JT for including the Tapestry numbers. The performance of it is just as good as I thought it would be. However, your description of Tapestry deserves a response. There’s no custom component for outputting the categories in the Tapestry test app, but the backing Java class is simply where the logic should reside. Tapestry has a built-in feature to reload page classes at development time (i.e. not hot code swapping, not context reloading) so this is just the way to do string concatenation in Tapestry – and I don’t think the same is true for all other frameworks. I mean, if you had a plain servlet implementation of the testbench, surely you wouldn’t argue it’s not valid because it’s implemented in plain Java?

    Requiring a custom component might improve the test, but not all the frameworks have “components” (in quotes since they do have similar concepts). Perhaps it’d be enough to just require generating a list of categories (i.e. with ul/li mark-up in between) to make it more valid?

    Whitespace compression is a feature of Tapestry, on by default. That can be controlled from the AppModule.java (see the source). Gzip compression is another built-in feature – I turned it off for this testbench to make things more equal compared to others. Finally, if you don’t mind updating the graphs, I’ll modify the testbench to use Tapestry 5.3 once we have the alpha out (likely not relevant in comparison to other frameworks, but I’m just interested to see how it performs against T5.2.5).

  81. @admin
    Could you pls add result for Play + standard Groovy template + Netty. Because its the most popular combination (standard combination), how Play! framework is used.
    Thank you!

  82. Please, add Grails + FreeMarker Plugin.

    See this microbenchmarking: http://techdm.com/grails/?p=445

    Very thanks for your work!

  83. I kind of wish that Kalle had done the Tapestry version using more typical Tapestry constructs: (a loop around a property expansion). I feel quite confident that there would be no perceptible difference in the performance numbers had he done so … quite literally, it’s the difference between a StringBuilder explicitly in the page class vs. a StringBuilder inside one of Tapestry’s internal classes (specifically, the Text DOM tree node).

    A more fair test would have been to output the list of Categories as an HTML list … even if I think Tapestry would suffer more here than other frameworks (creation of more DOM nodes). In any case, I would like to see that.

    Two more requests:

    I’d be very interested in some comparison of memory utilization.

    I also like to see an additional page that did perform some uniform amount of database access. I suspect you’d see some significant clumping of all the solutions once database access (along with the associated network activity) was incorporated.

    Thanks for putting together this test!

  84. @78 – admin

    Thanks for the update.

    For the particular scenario used in your tests, grails 1.3.7 with FreeMarker is performing better than grails 1.4 and jsp 2.1.

    @79 – admin

    Nice. Thanks!

    @83 – Wanderson

    Grails + FreeMarker added as grails-1.3.7-fm.

    For the interested reader:
    http://www.grails.org/plugin/freemarker
    http://www.grails.org/plugin/freemarker-tags

    Best regards,

    Daniel.

  85. Would have been interesting to see Vaadin in the comparison.

    Here is one study of Vaadin performance with 13.000 concurrent users generating 21.000 ajax requests per minute to one Tomcat server. http://vaadin.com/blog/-/blogs/vaadin-scalability-study-quicktickets

    http://vaadin.com/blog/-/blogs/vaadin-scalability-study-quicktickets

    (Yes, we used up to 14 EC2 large servers just to run JMeter to be able to simulate those users)

  86. […] claims to be performant and scalable. You want a proof? Than read this great post. The author compares the performance of Rails, Wicket, Grails, Play, Tapestry, Lift, JSP, […]

  87. FreeMarker should be even more fast on 1.4 due to the binding enhancements :)

  88. Soon groovyst/grailers will get (I think) similar response times to play-netty with Gretty.
    Gretty is netty for groovy++. Can’t wait netty includes the improvements Java7 brings to NIO with NIO2.

  89. I am currently relaunching a website from PHP to Tapestry5. Would be interested to see also a comparision to the YiiFramework.

  90. Any chance of seeing the python frameworks tested? I am most interested in pyramid, but there are plenty of others worth testing.

  91. What’s the version of Tomcat used in the tests and which connector is it configured to use (BIO, NIO, APR)?

  92. I’d like to see these tests applied to the last version of Play, specially with Play Scala. They have done some great improvements to the framework’s performance.

  93. Great benchmarks! It would be good to add Mono ASP.NET benchmark for further references.

  94. I’m interested on Wicket vs Tapestry comparison cause they are really similar, but I’ve download the code of both sample apps, and running 5000 on each one gave me:
    3 seconds responding 5000 for 1 user on Tapestry.
    4 seconds responding 5000 for 1 user on Wicket.

    I think may be there is a problem with the first graph on this page…. wicket is not responding as slower as it is mentioning… just a bit slower than Tapestry.
    Can someone else run these tests to compare the results too? And, for the author, please run again the test and update the graph on this page..

  95. Hi, thanks for the test. I’ve got a question if I may: did you warm up the JVM before the tests? I’ve read many complains of the tests not considering the impact of a warmed up JVM on the frameworks, and I would like to know if you did that as it’s not clear to me in the post.

    Thanks a lot!

  96. @Pere, 97
    I did a warm up in the following way. During benchmarking, JMeter shows the current response times including averages, deviations etc. When I start the benchmark, I watch the response time values. Depending on the framework and number of concurrent users and number of products, the response times can fluctuate a lot in the beginning, or might slowly drop. If the response time are stable after while, I perform a “clear” in JMeter to reset the averages and then start measuring. For some frameworks, I need to perform multiple clears so that I can monitor the latest stable averages. This way of benchmarking is more accurate than using default settings of Apache AB, which I have seen other people do.

  97. Please:

    1. Refresh Grails 1.4 to Grails 2.0.
    2. Set Grails 2.0 with FreeMarker Plugin
    3. (A MUST) New Graph using JDK 7 in all frameworks

    Thanks for all your work!

  98. Thanks for the tests!

    Is it possible to add the Apache Click[1] in the game?

    Regards,

    Gilberto
    [1] http://click.apache.org

  99. I hope this can be updated once Tapestry 5.3 is available; I’ve been working with some demanding clients and Tapestry is now about 30% faster under load … we’ve identified a bunch of thread-contention hotspots. Further, Tapestry 5.3 in production mode turns off live class reloading entirely, which reduces contention, memory utilization, and other factors quite a bit.

    Finally, I’d love to see a re-test as per the notes above; have all the frameworks list the categories as an <ul>/<li;> rather than a blob of text.

  100. Awesome test, but I’m not sure what grails-fm is.
    Super happy about play results, cant wait 2 ;)

  101. Its a freemarker:))

  102. Great benchmark, Thx.
    Since I like a lot Scala, Lift and Play! are the frameworks I am looking at and it’s nice
    to have such a performance test to compare the two.

  103. Hi,

    Can you add the new grails version? (grails 2.0)

    Thanks.

  104. How can you explain the huge performance difference between Play using Java an Play using Scala?

  105. Seels Lift and Wicket apps have problems that should be solved: http://apache-wicket.1842946.n4.nabble.com/Web-framework-performance-comparison-td3562647.html

  106. Romain,

    The difference between Play on Java vs Play on Scala is simple, they use different template engines.

    The default templating in the Java version uses Groovy, whereas the Scala version uses it’s own Scala based template engine that generates precompiled classes when building the site. The Scala version is more similar to the Japid template engine since Japid also generates code.

  107. I’ve found your benchmarks to be useful, however, I have also tried running these in Linux as opposed to Windows and have found that they run significantly faster in Linux. One reason for this is that unlike IIS which is designed to run on the kernel, these other web servers cannot run on the kernel in Windows and can’t get the best performance.

  108. Hello, I’ve just added a playapp + rythm template engine into your testing set. Rythm provides basically the the same level of performance with Japid (on my machine for 1000 product and 20 concurrent requests, the mean time of rythm is around 61ms, and Japid around 55ms) but much more easier to integrate with Play. The motivation for me to design this yet another static template engine is to create a Razor like syntax, fast, and easy to use tool. Migration from groovy to rythm is simple:

    1. rewrite your view file using rythm syntax (it’s very easy, trust me)
    2. annotate with your corresponding controller method with @UseRythmTemplateEngine when the view is changed to rythm.

    And that’s all. No need for play rythm:gen, and no need to rewrite your controller and make it extends RythmController. You can even reuse your FastTags in your application path without any expense.

    Check the source code and you will find how it works.

  109. Sorry, forgot to mention the pull request is here: https://github.com/jtdev/blogpost_files/pull/10

  110. From the test results it seems to indicate that Scala-Play is faster that Lift. Even in the high concurrency test it performed better.

    But this seems to be contrary to known data about the frameworks.

    Could it be that a wider variation of tests would tell a different story? [Lift vs Play wise I mean]

  111. Will you add test for play 2.0 ???

  112. @Jonathan What known data about the Play and Lift-Web frameworks are you referring to? Personally I’d actually expect Play to be a tiny bit faster

  113. I understand benchmarks are to be considered with a bit of salt. But, such microbenchmarks are informative neverthless. Thanks for the post. I am curious about adding Grails 2.0 and Play 2.0. They are both major rewrites over the previous versions. Could you make this public on Github or something so others can play with it.

  114. Never mind, just forked the github repo.

  115. I would love to see an update on this for Tapestry 5.3.4 (just about to be released), where I’ve put in some substantial performance improvements.

  116. Thank you for an extensive analysis of modern frameworks. I would like to add also my personal experience with Grails and Play: http://j2eespot.blogspot.com/2011/10/grails-vs-play-framework-comparison.html.
    Briefly, I tried both, I liked both in the beginning but like Grails more and more and Play less and less. Two major problems with Play for me:
    – Play 2 is not compatible with Play 1
    – Play is more and more about Scala and I found that I don’t like Scala

    Grails, on the other hand is an absolute pleasure to work with. Especially with help of Intellij IDEA.

  117. There is something wrong with this test. I’ve scaled rails on sites getting tens to hundreds of millions of requests per day using ruby and jruby, and the numbers you have above do not reflect reality. I don’t know what you are doing wrong, but something is wrong.

    As an example. Our xeon servers do around 700 requests per second with average response times of 20ms. And that’s with blocking calls to backend services like redis, mongo, memcache, etc..

    It’s easy to get just one thing wrong that can throw off all the numbers, especially across all these different frameworks where you might not have specific knowledge of how to configure them correctly.

  118. @Mikhail Gavryuchkov,

    – Play 2 is not compatible with Play 1
    You could stick with Play 1 and nothing is change right?

    – Play is more and more about Scala and I found that I don’t like Scala
    You could use Java in Play 2, again, nothing is change too.

    It’s seem like you were quoted from someone else comment to spread FUD.

  119. […] the built-in Play! template engine and delivers 3X as much total web throughput with Play!. See a third-party benchmark. The Japid change-reloading cycle is 5-10X faster than that of the Groovy/Scala based templates, a […]

  120. […] JT Dev published a benchmark for Java web frameworks and Ruby on Rails. The frameworks which have been compared are: […]

  121. […] Performance Comparison: Rails, Wicket, Grails, Play, Tapestry, Lift, JSP, Context […]

 Leave a Reply

(required)

(required)

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>