Saturday, August 22, 2009

Using memcache for Session Beans in GAE

Continued from Why Use JSP for GWT RPC Server-side Service

Note: the statement "JSP session beans are not supported in GAE" is outdated. Nonetheless, the examples on using memcache still holds. Session beans can be enabled in the appengine-web.xml:

<?xml version="1.0" encoding="utf-8"?>

[outdate info]
JSP session beans are not supported in GAE. The jsp:usebean tag has no effect of value in a JSP running in GAE. As session control is not implemented in GAE, we would need to exploit persistence features of GAE to implement it. A strategy would need to be employed to put and get session beans. The tactic described here uses memcache, which is implemented by GAE as standard Java caching API (import javax.cache.*).
[/oudated info]

GAE memcache is not actually session cache by itself because the stored blobs do not actually expire. We have to deliberately set an expiry time.

SessionSilo, below, can be used as a template for implementing session persistence of beans. Please refer to synthful project in Google Code for its complete source code. Here is the outline of the strategy.
  1. At start of session initialise persistence cache.
  2. Set an expiry time for the cache.
  3. At the first JSP response within a session, declare the bean.
  4. At every start of JSP response, get the session bean from persistence cache.
  5. At end of JSP response, if the bean had updates, put the bean into persistence cache, overwriting its old blob stored in the cache.

package com.blessedgeek.gwt.gdata.server;

public class SessionSilo

/** * Create new instance of static cache, beanCache. * GAE memcache is configurable thro certain reserved keys. * A reserved key and its value can be injected into the * cache thro a hashmap. * If a config hashmap is not injected into the cache, * the cache would take on default values. * */ static public void initBeanCache(){ if (beanCache!=null) return; Map cfgMap = new HashMap(); cfgMap.put(GCacheFactory.EXPIRATION_DELTA, 900); try{ CacheFactory cacheFactory = CacheManager.getInstance().getCacheFactory(); beanCache = cacheFactory.createCache(cfgMap); } catch (CacheException e){} }
/** * Call this at start of a JSP to get the session bean * If bean does not exist, create and put it in the cache. */ static public MrBean initSessionBean(String sessId){ MrBean mrBean = getBean(sessId); if (mrBean==null){ mrBean = new MrBean(); mrBean.sessionId = sessId; putBean(sessId, mrBean); } return mrBean; }
/** * Call this at the end of a JSP. * If any routine updates the bean, * those routines would need to set updated flag to true. * If updated is true, then put the bean back into cache * to over-write its existing blob in the cache. * * @param mrBean */ static public void storeSessionBean(MrBean mrBean){ if(mrBean.isUpdated()) putBean(mrBean.sessionId, mrBean); }
static Cache beanCache; }

Example of JSP using SessionSilo static methods to store and get "session" beans.
page language="java"
MrBean mrBean = SessionSilo.initSessionBean(session.getId());


In the TableMgr application, TableActionService.gwtrpc.jsp employs this technique.

Its subsidiary JSPs conditionally called-by-include also have to employ these series of manoeuvres because GAE does not pass bean instances across an include call: Listxxx.jsp, Loggedin.jsp.

Remember that a bean being cached must implement the java.util.serializable interface. Every member class of the bean as well as the whole depth of recursive members must also implement serializable. Otherwise, GAE will dump the response due to encountering non-serializable exception when attempting to put the bean into memcache.

If a bean has many members, it may be better to cache individual members of the bean. Why should a whole cached blob be replaced if only one tiny bit was changed for each response? Caching individual members of a bean also allows a bean to have non-serializable members that do not need persistence.

You should also read

Why Use JSP for GWT RPC Server-side Service

Continued from GWT RPC - How Server-side JSP Service Works

Because, you can produce json response this way.

<%@ page
contentType="text/html; charset=utf-8"
MrBean mrBean = SessionSilo.initSessionBean(session.getId());
TableEntry entry = mrBean.getTableEntry();
Data data = entry.getData();
 int i=0;
 for (Column col : data.getColumns())

Because, that is why programmers prefer to use PHP, Perl or Scala to produce dynamic web pages rather than using a non-JSP servlet.

Because this is ugly and unfriendly, ugly to copy n'pasting (excerpted from
if (user != null){
  if (user.is_facebook_user()){
    html += "<div id=\"header-profilepic\">";
    html += user.getProfilePic(true);
    html += "</div>\n";

  html += "<div id=\"header-account\">";
  html += "<b>Welcome, " + user.getName() + "</b>";
  html += "<div class=\"account_links\">";
  html += "<a href=\"account.jsp\">Account Settings</a> | ";
  if (user.is_facebook_user()){
    html += "<a href=\"#\" onclick=\"FB.Connect.logout(
    )\">" +
    "Logout of Facebook" +
    //.'<img src="images/fbconnect_logout.png">'
    html += "<a href=\"logout.jsp\">Logout</a>";
  html += "<br /></div>";
  html += "</div>\n";
  html += "<div class=\"account\">";
  html += "Hello Guest | ";
  html += "<a href=\"./register.jsp\">Register for an account</a>";
  html += "</div>\n";

html += "</div></div>\n"; // header & header_content
html += "<div class=\"body_content\">\n";

Because, you can use my text custom tag ( to subject large blocks of unbroken text to conditional output.

Continue next Using memcache for Session Beans in GAE

GWT RPC - How Server-side JSP Service Works

Continued from Passing Parameters in GWT RPC (Server-side)

Remember that the name of any JSP that is registered as a GWT RPC service responder must end with suffix ".gwtrpc.jsp".

Here's the reason why.

A JSP is first translated into a Java source file with a _jspService method, which always begins with this block of code
public void _jspService(
HttpServletRequest request,
HttpServletResponse response)
throws, ServletException {

JspFactory _jspxFactory = null;
  PageContext pageContext = null;
  HttpSession session = null;
  ServletContext application = null;
  ServletConfig config = null;
  JspWriter out = null;
  Object page = this;
  JspWriter _jspx_out = null;
  PageContext _jspx_page_context = null;

  try {
    _jspxFactory = JspFactory.getDefaultFactory();
    pageContext =
    this, request, response,
    null, true, 8192, true);
    _jspx_page_context = pageContext;
    application = pageContext.getServletContext();
    config = pageContext.getServletConfig();
    session = pageContext.getSession();
    out = pageContext.getOut();
    _jspx_out = out;


However, the RemoteServiceServlet class, which a GWT RPC service responder servlet should extend (in order to exploit the convenience of data serialization afforded by implementng GWT RPC interfaces), uses its own output stream and ignores/discards the response output stream. That is, the JspWriter out stream is ignored/discarded. Which causes the output of a JSP extending RemoteServiceServlet to be blank, because all the content rendering of a JSP is translated into lines of ignored output stream

out.write( .... );

Therefore, we need a way to exploit the out.write lines generated by the JSP translator, by diverting out back to the response body. So we create a class JspResponseWriter, which extends JspWriter, to over-ride the write method, so that it writes to the response body.
package org.synthful.gwt.http.servlet.server;
public class JspResponseWriter
 extends JspWriter

@Override public void write(char[] cbuf, int off, int len) throws IOException{ for (int i=off; i<off+len; i++) this.Body.append(cbuf[i]); }
.... }

Alright, now we the mice have the bell, we only need find a ribbon to tie it to the neck of the pussycat. Even with a ribbon, we still need to sneak up to the cat tie the ribbon with the bell around the cat.

Okay, notice in the translated JSP Java source is the line

out = pageContext.getOut();

So we create an extension of PageContext whose getOut method would sneak our version of JspWriter to the out variable.

And, the plot thickens. Now, we have to find a way to sneak our version of PageContext into the conspiracy to defraud Google's blindfolding of the JSP out prisoner.

Notice the translated Java source has a line
pageContext = _jspxFactory.getPageContext(....);

And your premonition is correct - we have to send in a double-agent to sneak our impersonator of the JspFactory, who would help us sneak our version of PageContext into the belly of RemoteServiceServlet.

So, here's the double-agent, JspServiceBeanable, who would undertake this mission. Notice the line JspFactory.setDefaultFactory( .... ), which is how our agent sneaks our version of JspFactory, which would sneak our version of PageContext, which would sneak our version of JspWriter, which would sneak GWT server-side prisoner, the JSP rendering output, back to the response body where it rightly belongs:
package org.synthful.gwt.http.servlet.server;

abstract public class JspServiceBeanable
  extends RemoteServiceServlet
  implements HttpJspPage

protected String doJspService( HashMap<String, String> parameters){ this.parameters = new Hashtable<String, String>(parameters); JspFactory.setDefaultFactory(new JspFactoryShunt()); this.jspFactory = JspFactory.getDefaultFactory(); try{ this._jspService( this.getThreadLocalRequest(), this.getThreadLocalResponse()); } catch (Exception e){ e.printStackTrace(); } this.jspContext.popBody(); return this.jspOut.Body.toString(); }
public class JspFactoryShunt extends JspFactory { .... @Override public PageContext getPageContext( Servlet arg0, ServletRequest arg1, ServletResponse arg2, String arg3, boolean arg4, int arg5, boolean arg6) { Class cls = arg0.getClass(); this.shuntedPageContext = this.shuntedJspFactory.getPageContext( arg0, arg1, arg2, arg3, arg4, arg5, arg6); if (!cls.getName().contains("_gwtrpc_jsp")) return this.shuntedPageContext; jspContext = new PageContextShunt(this.shuntedPageContext); return jspContext; } .... }
public class PageContextShunt extends PageContext { public PageContextShunt(PageContext ctx) { this.shuntedPageContext = ctx; } .... }

But, but, but ...,
JspFactory.setDefaultFactory and
are static methods, and affects the whole application, not just a request, so that our version of JspFactory is thrusted into the throats of even JSPs who are not involved in this conspiracy.

We need to a way for our JspFactory impersonator to differentiate between visitors and prisoners, between JSPs not part of the crime from JSPs intent on committing this fraud. Otherwise, the output of the non-complicit JSPs would not be processed as intended thro the normal response out stream.

That is where the suffix, .gwtrpc.jsp comes in. Notice the code in the JspFactoryShunt, which returns the original shunted PageContext for JSPs whose names are without .gwtrpc.jsp.

if (!cls.getName().contains("_gwtrpc_jsp"))
  return this.shuntedPageContext;

Continue next Why Use JSP for GWT RPC Server-side Service

Tuesday, August 11, 2009

Passing Parameters in GWT RPC (Server-side)

Continued from Passing Parameters in GWT RPC (Client-side)

The server-side code
The server-side servlet must extend RemoteServiceServlet, remember?
However, to allow a JSP to be the servlet servicing the RPC request, the JSP must implement the HttpJspPage interface. JspServiceBeanable, which does that, is an abstract class (provided by the courtesy of this author) to be extended by a JSP.

package org.synthful.gwt.http.servlet.server;

abstract public class JspServiceBeanable
extends RemoteServiceServlet
implements HttpJspPage

protected String doJspService( HashMap<String, String> parameters) { this.parameters = new Hashtable<String, String>(parameters); JspFactory.setDefaultFactory(new JspFactoryShunt()); this.jspFactory = JspFactory.getDefaultFactory(); try{ this._jspService( this.getThreadLocalRequest(), this.getThreadLocalResponse()); } catch (Exception e){ e.printStackTrace(); } this.jspContext.popBody(); return this.jspOut.Body.toString(); }
.... }

But, wait another minute. Shouldn't it also implement HashedParameterService in order to recognise the HashMap datatype being passed by the RPC requester?

Wokay dokay, here's JspServiceParametricBeanable, the abstract class to be extended by any JSP with ambitions to service an RPC request that comes with a HashMap argument. It over-rides the empty doServiceResponse method in GWT-supplied RemoteServiceServlet so that instead of ignoring the response output, it diverts RemoteServiceServlet to perform doJspService mandated by HttpJspPage interface.

package org.synthful.gwt.http.servlet.server;
import java.util.HashMap;
abstract public class JspServiceParametricBeanable
extends JspServiceBeanable
 public String doServiceResponse(
  HashMap<String, String> parameters){
  return this.doJspService(parameters);

Oopsie doops, wait one more final minute. Shouldn't the JSP implement TableMgrService which the application-specific client-side in the previous page has required to define the target URL? So here's the actual abstract class to be implemented by the RPC service JSP.
package com.blessedgeek.gwt.gdata.server;
import com.blessedgeek.gwt.gdata.client.TableMgrService;

abstract public class TableMgrServiceImplJspBeanable
 extends JspServiceParametricBeanable
 implements TableMgrService{}

So here's the header for TableActionService.gwtrpt.jsp as targeted by TableMgrService interface. Notice that it extends TableMgrServiceImplJspBeanable.

<%@ page


When using JspServiceBeanable as the means to allow JSPs to be used as GWT RPC service responders, the servicing JSP name must end with ".gwtrpc.jsp". JspServiceBeanable detects that a JSP is a GWT RPC service responder by detecting if its name ends of ".gwtrpc.jsp".

Continue next GWT RPC - How Server-side JSP Service Works

Passing Parameters in GWT RPC (Client-side)

Continued from GWT RPC

The GWT RPC tutorials provide the simple case of passing and receiving String objects. However, in most, if not all, practical cases, programmers need to pass a set of parameter-value pairs between HTTP communicants. In this case, between communicants of the communion of RPC over HTTP.

TableMgr bears in itself an example how to pass parameters over the RPC/GWT/HTTP (RPC over GWT over HTTP) communion.

For TableMgr source code, refer to Table Manager,

and for the library elements it uses, refer to

Do not copy code from this page, as it would contain line-breaks and exclude lines left out for brevity that would break the code, but get them drectly from the google code project listed above.

TableMgr uses a HashMap as the datatype being passed. A HashMap is the best choice because it allows values to be mapped to parameter keys.

GWT client-side code
package org.synthful.gwt.http.servlet.client;
import java.util.HashMap;

public interface HashedParameterService
extends RemoteService
 String doServiceResponse(
  HashMap<String, String> input);

The interface HashedParameterService is a generic service declaration interface (provided by the courtesy of this author) which anyone could use, but if you use it with GWT-provided server side RemoteServiceServlet, your servlet would need to provide the method

String doServiceResponse(
 HashMap<String, String> input);

GWT applications in accessory to using this framework is restricted to having this method as the RPC procedure being called.

Or, you could copy it and rename the method to a name of your liking. However, my intention for it was not to provide a generic service declaration interface, but for a more sinister intention. It is to subvert the RemoteServiceServlet provided by GWT so that I could have JSPs, rather than plain servlets, implement it to service RPC requests. This would be explained later in the server-side part.

Wait a minute, shouldn't the service declaration interface contain the RemoteServiceRelativePath directive to tell the GWT compiler the URL that would service its RPC request? Therefore, to keep this interface generic, we introduce an extra layer of interface.
package com.blessedgeek.gwt.gdata.client;
public interface TableMgrService
 extends HashedParameterService

TableMgrService would be the actual interface implemented by the RPC client, but the server side would still implement its super-interface HashedParameterService. The corresponding client and server sides do not actually have to implement the same interface but interfaces of the same signature.

The companion async-response request interface to that is

package org.synthful.gwt.http.servlet.client;
import java.util.HashMap;

* @gwt.TypeArgs parameters <java.util.HashMap 
public interface HashedParameterServiceAsync
 void doServiceResponse(
  HashMap<String, String> parameters,
 AsyncCallback<String> callback);

Note that both of them declares the same HashMap<String, String> parameters argument.

Continue next Passing Parameters in GWT RPC (Server-side)

Monday, August 10, 2009


Continued from TableMgr - List Spreadsheets

GWT is an ingenious way of allowing browser behaviour to be coded in Java source code, which the GWT compiler then compiles into Javascript.

GWT could allow the programmer to escape from Java into writing Javascript using the Java native method declaration. As far as GWT is concerned, Javascript is the native code of the browser - even though to the operating system, that is far from the truth.

Bear in mind not to confuse the Java source code for the browser with the Java source code running on the web application server. The Java source code for the browser is compiled into Javascript, whereas the Java source code on the server is compiled into Java bytecode. Of course, the server code need not be coded in Java, as long as it provides the GET-able URLs that a GWT client page requests.

GWT RPC allows a page to request for content from its server using the established HTTP channel. The GWT Java source (before compilation to Javascript) mimics a remote procedure call. However, the RPC is actually translated into normal HTTP requests/responses. Perhaps, just as many protocols ride on HTTP, we could call this RPC over HTTP. XYZ over HTTP is a common strategy programmers use to allow information to mimic web pages, to circumvent restrictions placed by internet traffic policing devices on non-HTTP communication.

Second Level Domain Same Origin Policy (SLD-SOP)
Bear in mind that the standard browser restricts its pages to only be able to request for, and to include, content from within the same second-level domain (SLD) as the server that provided that page to the browser. Top level domains (and top-level and a half) are such as .com, .org, .net, or So, domain names such as,, are second level domains. While, would be a third-level domain. Therefore, GWT RPC would work only within the confines of an SLD.

Note that SLD-SOP also requires the port number to be the same, such that is not on the same SLD as A particular browser brand may be lax and allows different port numbers within the same SLD consideration, but that would not be compliant to standard SLD restrictions.

GWT client-side code
The Java source of the GWT client intending to conduct RPC with its server must provide its own service declaration interface,
  • which has a @RemoteServiceRelativePath directive to tell the GWT compiler the URL that would service its RPC request.
  • which declares the remote service method to tell the GWT RPC class the datatype of the request argument and the expected return datatype. Surely, you need to pass arguments when you call a procedure and expect an object to to be returned. The datatype of the argument and the return signature and all their recursive members must implement the serializable interface (of course, or expect a nonserializable run-time error when the RPC process attempts to serialize those data).
  • where the remote service method is to be invoked on the client-side GWT code to trigger the RPC.
  • where the name of remote service method corresponds to the name of the remote method/procedure that GWT RPC would pass to the server to invoke to service this call.
  • which must implement the interface, which is an empty, marker interface.
The service declaration interface is a bridging interface between the client and the server as it is used by both the client and server sides. It is to be implemented and invoked by the GWT class used for performing the RPC on the client side as well as implemented and invoked by the servicing servlet on the server side.

The GWT client source code must also provide an async-response request interface for the GWT compiler. This interface should declare a method with two arguments
  • the serializable datatype declared in the service declaration interface, to tell the GWT compiler constructing this whole setup the datatype of the request argument and the datatype of the object to be returned by the RPC service.
  • the call-back method of the client that would asynchronously handle the server response. The call-back method must implement the interface.

The server-side code
The URL on the server-side targeted by the GWT RPC client must, of course, be able to recognise the request parameters. How else would it be best but for the servlet to implement service declaration interface provided by the client. For a server that implements JEE standards - that is obvious. However, if the responder on the server is not written in Java - it would have to mimic the serialization of Java datatype declared in that interface. Since this is not a hitch hiker's guide to PHP, Python or Perl, etc, non-JEE responders are beyond the scope of discussion.

Remember that the service declaration interface defined the remote service method which your server is "supposed to" use to service the RPC request. You could write your own RPC request handler servlet by ignoring the service declaration interface as well as the remote service method name it declares but service the request any old how. Your handler servlet would still need to deserialize the RPC request parameters.

If you are coding server-side in Java, GWT provides the server-side request handler as well, RemoteServiceServlet, to
  • call the method as requested by the GWT client. The method must implement service declaration interface which you have declared for the client.
  • deserialize the data, that came with the request, back into a Java object, and pass the object to the requested method.
  • serialize the return type of the method to be sent back to the GWT client.
Which is a lot of work not to write your server side in Java.

Naturally, your handler servlet would need to be mapped to the URL on which the GWT client has sent a GET request, as registered in the @RemoteServiceRelativePath directive of service declaration interface.

To use the GWT framework, your servlet needs to extend

Continue next Passing Parameters in GWT RPC (Client-side)

Wednesday, August 5, 2009

TableMgr - List Spreadsheets

Continued from Cuckooberra Login Process

The following diagram illustrates the List Worksheets action which would exemplify the rest of the information flow between browser, Cuckooberra GAE server and Google Docs GDATA service.

Continue next GWT RPC

Tuesday, August 4, 2009

Cuckooberra Login Process

Continued from TableMgr: GAE + GWT + GDATA with RPC

The following diagram documents the log-in procedure.

A one-time-use authorisation token is dispensed by Google accounts when you log in. This one-time-use token is then exchanged for a session token. When you log out of Cuckooberra, the session token is revoked. Remember that the session token actually never expires. So if you logged in to try Cuckooberra, and any interruption occurs preventing you from loggin out, there is a revoke token link at the left bottom of the page which you should click to revoke the token. Cuckooberra does not memorise the token beyond the browser session, so it would not be able to remember what token you used if you return to use it after the interruption to your browser session.

Note: "Redirect to Google Accounts" in Login.jsp is not really a HTTP redirect but a Javascript "location.replace()".

Continue next TableMgr - List Spreadsheets

TableMgr: GAE + GWT + GDATA with RPC

TableMgr is a web application hosted on Google App Engine. It is actually a port of Google's spreadsheet and table GDATA examples from being a console application to running as a GAE/GWT application.

Therefore, it is a combination of Google App Engine, Google Web Toolkit and Google Spreadsheet access through GDATA service.

It also makes use of GWT RPC to avoid the need for page refresh. I also extended the Http servlet framework that GWT RPC service uses so that I could use JSPs to respond to RPC service requests.

The code is found here:

The library elements it uses are found among here:

TableMgr resides within the GAE application ID Cuckooberra. The following page documents the Cuckooberra log-in process.

Continue to Cuckooberra Login Process

Sunday, August 2, 2009

Why I love to scrum 2

Continuing from my last blog about misplaced agile efforts, I should write, for the record, that I had always enjoyed agile and scrumming process for my other jobs.

Agile development and scrumming practices are useful only when they are not constrained by formalised theories.

I used to work with a cross-location, cross-department project team which had broad support from the departments that were directly involved in making the company profitable. It is great to work for a team supported by a user-base who have the financial authority and direct motivation to make the company profitable. They don't care about information management theories or violation of theories people had learnt from MBA or MISc degrees. They just want to make sure you are helping them make money for the company. I like working for such clients. They don't have the luxury for adhering to information management theories and most of all, they have the authority and discipline to over-ride those who do.

We scrummed by email, Solaris talk and telephone and sometimes video-conf. We only got those who had the problems and solutions involved. We were careful not to waste anybody's time. We made regular enhancements and releases. We had two weeks of beta for every release in which advanced users would test the new releases. We had an effective release roll-back procedure. It was a rather large multi-national and so we had to manage with different sites having different releases. There was a corporate project manager who did his job effectively. We had fluid project milestones and a queue of feature requests and a feature priority system. The feature prioritisation process coincided with what we would today call the scrum of scrums.

Now and then, some of my Information Department colleagues would berate my team as moving targets illegitimately existing on the absence of formal procedures and that our team simply survived only because we had the profits generating people behind us.

Hello? Only because we had the money generating people behind us? Why would the money generating people be behind us, in the first place?

Ok, since we were a large multinational, we tend to have more than one solution. A competing team decided to steal me away from my first team and the relocaton benefits were wonderful. I succumbed to the offer. It was another great team.

We scrummed as often as we needed resolutions. It was more like extreme engineering.

Once, a team member noticed a persistent spike in the development Sybase server that lasted for more than ten minutes. Two members of the team traced the source to a piece of code with cross-product query. A quick scrum was called and a raise of hands was made to find out who else was writing cross-products. Immediately, we had a microproject of excising all cross-product queries from our our application. We traced all the code to one single culprit who was a contractor. I just don't understand, none of the other team members did, why would he persist in writing cross-product queries after the team had decided to ban it. Doesn't he understand what 2.5 normalisation means? We were not in the business of sticking to normalisation perfection but to ensure our application ran as fast as possible. OK, we decided not to renew his contract because he kept citing importance of normalisation theory over performance.

We had a colleague, who was not in the team, who had decided that we lacked formal discipline and decided to help us rectify that shortcoming. He wrote a Delphi application that would help us track our activities to the time spent on each module of the application to help us justify if a module was worth the effort. All we had to do was, keep his application up all the time and key in whenever we started or stopped our work on each respective module. He shared it with the division head who thought it was a great idea. It was a riot because most of us (unlike he) were multi-taskers. The project manager managed to persuade our division head that it was a bad idea, fortunately. Our colleague was extremely disappointed because we had caused his efforts to be wasted. We were using AIX workstations and he thought all we needed was the simple case of running the PC emulation window to keep his application alive. OK, he thinks we have the CPU power to spare for a PC emulation window. I had only used the PC emulation window at the end of the day to play computer pong.

I just don't understand why now and then, there are colleagues who work jealously against an agile development team.

My next pleasant experience with agile development was in Colorado as a 1099 contractor. My client and I formed an extreme programming team. It was like playing ping-pong with him, sitting in the same cubicle. He was a software product manager with no one reporting to him, except a couple of contractors like me. Moreover, the view of the Rockies were great. Once, he drove me to South Park and declared that I was in an historic town. At that time, I had not realised that South Park, Colorado was the raison d'etre for the South Park cartoon. Until he told me. I mean, how could evangelical Colorado be the origins of such an irreverent cartoon show?

Then, I decided to return to the east coast, to the dismay of my Colorado client where I worked for a privately held company. It was a wonderful development environment. My management measured me by the level of happiness of my clients. They reminded me that I was there to provide statistical solutions and warned that it was a fluid environment. Finally an IS department that understands what profitable means. I am sure, being a small company makes profit-consciousness a top priority for the IS director. What a pleasant surprise. I had great reviews. When I had to live in a neighbouring state for personal reasons, they sought to retain me to accommodate my need of having to drive 6o miles to work until I found another job.

Saturday, August 1, 2009

Why I love to scrum

Before, scrumming and agile development approach was formalised with the technical terms "Agile Engineering", my colleagues and I were already practising it.

We had regular scrumming sessions over Solaris talk and telephone conference as well as face to face informal meetings. As agile development adage goes, we made application changes precise, frequent and quick, so that users could enjoy the fruits of our modifications as soon as possible. At every release we had two weeks of beta for willing users who wanted to use the latest and greatest or for users who themselves urgently required a feature they had suggested because a disastrous process misbehaviour had slipped through their fingers which the new statistical analysis features would catch.

The management of the users were very supportive and their mandate was make my engineers, planners and designers happy. That was since twenty years ago. Yes, I have been practising scrumming and agile development for twenty years with my colleagues but we never knew that one day what we were practising would be formalised into terminologies called scrumming, extreme engineering and agile development.

In one of my former place of employment, since I normally reported to the Information Systems department, I had a problem. My administrative management would inform me that the project goals and targets that I had agreed with my manager were seldom satisfied. So, came the annual goal and target setting week.

What are my project goals and targets for this coming year, I would be asked. I would sit quiet, pondering how to transform my tasks into projects and milestones that would be acceptable to my manager. Finally, I would confess, I really don't know. What? My goal is to ensure that the users of my statistical analysis applications are happy with my work. That is not a valid goal, I would be told. OK, I proposed, how about goals like these,
  • reduce average turnaround times of tasks from two weeks to ten days
  • reduce number of misinterpreted customer requests from five to one.

Those are not project goals.

But, but, but, my clients' goals and targets keep changing. I don't know what they want next. They seldom know what they want next.

You must force your customers to accept the way the Information Department works. We have solid projects not moving targets. We should have customer contracts which we would fulfill and which the client department needs to justify through their department managers why those targets and goals need to change. You are working for the Information Department, remember? Not for Engineering, Planning, Manufacturing or Design. You must force your departmental clients to understand how information management works. If you fail to make your clients understand and comply to that, we are sorry to tell you that you have no skills in managing your customers.


But my departmental clients are in the business of making their customers happy. Moreover, I together with every employee, have been compelled by the corporate quality compliance authorities to pledge through various fanciful sloganeering that we would work our best to keep our corporation profitable by helping our colleagues ensure our external customers are satisfied.

So, why can't I have a set of stable immutable projects? That is because, I work for a corporation which has processes. Both procedural and chemical processes that need to be managed. New products and processes are constantly developed. My departmental clients cannot wait for a process to stabilise into immutable states because if we do not solve issues as soon as they arise, our external customers would take their contract elsewhere. It's a fast changing business. We have to constantly stabilise those processes by constantly discovering requirements for information that we never expected we needed last week. Therefore, my targets keep changing and therefore I cannot have long term project milestones. My milestones are one week as soon as a new issue crops up and I would need to scrum with Engineering, Manufacturing, Planning and Design to remove bottlenecks and roadblocks towards the goals of making our external customers happy.

Then after a while, my psychotic management would decide - so we need to justify your existence because they have just graduated with MBA degrees where they learnt that we need measurable benefits in what we do. If we cannot measure, we are probably doing the wrong thing.

OK, why don't you get your departmental customers tell you how much money they saved for every microproject they make you perform? We have an activity-based costing framework and we need to distribute the cost of employing you to various product lines and the cost saving differentials. So, I spent two days conjuring numbers with my clients, instead of helping them solve problems. After, just one round of cost justification exercise, my colleagues implied that I should either work on their problems or f*** off.

So, on the whole, sometimes information departments are really out of touch with their fellow colleagues on the nature of the business of the company they work for. They are so attached to their information theories and formalised information management theories that they seem to think that the whole world needs to tune their processes to be in line with computer science.

Then, one day, I started working for a company that prided itself with formalised scrumming practice.

Hey, fellow agile development enthusiasts, the day scrumming became formalised is the day of its demise. When I participate in a daily scrum, this is what I expect to happen. I want to present to my colleagues the roadblocks I am facing. Then I want to find out who has the expertise, authority or personnel connections to remove my roadblocks and I would like to find out if I am roadblock to my colleagues and what I can do about that and also if I am the solution to helping someone's roadblocks. And I expect everyone on the team to bear the same attitude. Then I expect that we humbly acknowledge we need to rearrange our fluid milestones and schedule which is subject to changes. Then I expect the product to be tested and then released to customers or customer interface who are willing to provide feedback to the product.

However, the department that I started to work for did not realise they had quite a few impediments to their successful implementation of agile development. Their daily scrum lasts an hour. People took turns to present an update of their tasks and justify their delays. And the scrum was presided over by the scrum-enthusiastic department head, no less. Alright, after those meetings, I would have to go through the actual scrumming by meeting informally with colleagues which is when I get my real scrum done.

So, the company was experiencing frequent customer returns. These are huge pieces of equipment not little tubes of IC chips. Or the service engineer would have to drive or fly a couple of thousand miles to see what the problem is with the equipment. The idea of agile engineering, I was reminded was to work on an issue, and don't try to solve the bigger picture and refrain from refactoring your code. But that advice works only after a while. With the frequency of customer returns, the only solutions was to clean up the code that I had inherited because patch after patch was making it difficult to trace problems.

The other problem was, customer interface personnel who worked with us refused to participate in scrums. In order to find out what the problem was, we needed to make changes and have her try it out. But she complained to the department head that she was confused by my frequently updating the behaviour of the application. Ok, she complains the customer is not able to use the equipment because of software problems. I am told that the formalised theory of agile development is not to make large changes. And without making large changes I cannot clean up the code. Without cleaning up the code, the patches upon patches made it impossible for me to predict if when released, would the equipment be returned. And to clean up the code I would need our gentle and lovely customer interface to understand that she would need to test them.

It is unfortunate when once useful and practical procedures are formalised into respectable MBA curriculum, they normally become the problem rather than solutions. People don't know when to deviate from the theory.

As for me, I just want to make my customers happy.