If you have your web application running on one tomcat instance and want to add another tomcat instance (ideally on a different machine), following steps will guide you.
Step 1: Independently deploy your web application (WAR file) on each instance and make sure they can work independently.
Step 2: Stop tomcat
Step 3: Update the <Cluster> element under the <Engine> element in the Server.xml file (under the conf dir in tomcat installation dir) on both your servers with:
Step 4: Start tomcat and make sure it starts up correctly. You should be able to access http://locahost:8080. Most default tomcat installations come with an examples web app. Try access http://localhost:8080/examples/jsp/ You should see a list of JSP files.
Step 4.a: Also if you see catalina.out log file, you should see:
INFO: Initializing Coyote HTTP/1.1 on http-8080
Nov 9, 2009 9:29:43 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 762 ms
Nov 9, 2009 9:29:43 AM org.apache.catalina.core.StandardService start
INFO: Starting service <server_name>
Nov 9, 2009 9:29:43 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.16
Nov 9, 2009 9:29:43 AM org.apache.catalina.ha.tcp.SimpleTcpCluster start
INFO: Cluster is about to start
Nov 9, 2009 9:29:43 AM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:/<server_ip>:4000
Nov 9, 2009 9:29:43 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to 500
Nov 9, 2009 9:29:43 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4
Step 5: Stop tomcat.
Step 6: We’ll use the examples web app to test if our session replication is working as expected.
Step 6.a: Open the Web.xml file of the “examples” web app in your webapps. Mark this web app distributable, by adding a <distributable/> element at the end of the Web.xml file (just before the </web-app> element)
Step 6.b: Add the session JSP file. This JSP prints the contents of the session and also adds/increments a counter stored in the session.
Step 6.c: Start tomcat on both machines
Step 6.d: You should see the following log in catalina.out
Nov 9, 2009 9:29:44 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 0, 101}:4000,{-64, -88, 0, 101},4000, alive=10035,id={68 106 92 39 -110 -8 73 124 -116 -122 -15 -3 11 117 56 105 }, payload={}, command={}, domain={}, ]
Nov 9, 2009 9:29:49 AM org.apache.catalina.ha.session.DeltaManager start
INFO: Register manager /examples to cluster element Engine with name <server_name>
Nov 9, 2009 9:29:49 AM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /examples
Nov 9, 2009 9:29:49 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
WARNING: Manager [localhost#/examples], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{-64, -88, 0, 101}:4000,{-64, -88, 0, 101},4000, alive=15538,id={68 106 92 39 -110 -8 73 124 -116 -122 -15 -3 11 117 56 105 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.
Nov 9, 2009 9:29:49 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Manager [localhost#/examples]; session state send at 11/9/09 9:29 AM received in 101 ms.
Nov 9, 2009 9:29:49 AM org.apache.catalina.core.ApplicationContext log
INFO: ContextListener: contextInitialized()
Nov 9, 2009 9:29:49 AM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: contextInitialized()
Nov 9, 2009 9:29:50 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8080
Nov 9, 2009 9:29:50 AM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on /0.0.0.0:8009
Nov 9, 2009 9:29:50 AM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/49 config=null
Nov 9, 2009 9:29:50 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 6331 ms
Step 6.e: Try to access http://localhost:8080/examples/jsp/session.jsp Try refreshing the page a few times, you should see the counter getting updated.
Step 6.f: You should see the same behavior when you try to access the other tomcat server. Open another tab in your browser and hit http://<other_server_ip>:8080/examples/jsp/session.jsp
Step 6.g: At this point we know the app works fine and the session is working correctly. Now we want to check if the tomcat cluster is replicating the session info. To check this, we want to pass the session from server 1 to session 2 and see if it increments the counter from where we left.
Step 6.h: Before accessing the page, make sure you copy the j_session_id from server 1 (displayed on the http://localhost:8080/examples/jsp/session.jsp). Also make sure to clear all cookies from server 2. (All browsers give you a facility to clear cookies from a specific host/ip).
Step 6.i: Now hit http://<server_2_ip>:8080/examples/jsp/session.jsp;jsessionid=<jsession_id_from_server1>
Step 6.j: If you see the counter incrementing from where ever you had left, congrats! You have session replication working.
Step 6.k: Also catalina.out log file should have:
Nov 9, 2009 9:42:03 AM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: sessionCreated('CDC57B8C5CFDFDDC2C8572E7D14C0D28')
Nov 9, 2009 9:42:03 AM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeAdded('CDC57B8C5CFDFDDC2C8572E7D14C0D28', 'counter', '1')
Nov 9, 2009 9:42:05 AM org.apache.catalina.core.ApplicationContext log
INFO: SessionListener: attributeReplaced('CDC57B8C5CFDFDDC2C8572E7D14C0D28', 'counter', '2')
While this might like smooth, I ran into lot of issues when getting to this point. Following are some trap routes I ran into:
1) java.sql.SQLException: No suitable driver tomcat cluster
Make sure your DB Driver jar (in our case mysql-connector-java-x.x.xx-bin.jar) is in tomcat/lib folder
2) In catalina.org if you see the following exception:
Nov 7, 20093:48:53 PM org.apache.catalina.ha.session.DeltaManager requestCompleted
SEVERE: Unable to serialize delta request for sessionid [1F43C3926FF3CC231574EF248896DCA6]
java.io.NotSerializableException: com.company.product.Class
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1156)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at java.util.ArrayList.writeObject(ArrayList.java:570)
at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
This means that you are storing com.company.product.Class object (or some other object that holds a reference to this Object) in your session. And you’ll need to make com.company.product.Class implement Serializable interface.
3) In your catalina.out log if you see
INFO: Register manager /<your_app_name> to cluster element Engine with name <tomcat_engine_name>
Nov 7, 2009 11:56:20 AM org.apache.catalina.ha.session.DeltaManager start
INFO: Starting clustering manager at /<your_app_name>
Nov 7, 2009 11:56:20 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [localhost#/<your_app_name>]: <strong>skipping state transfer. No members active in cluster group</strong>.
If both your tomcat instance are up and running, then check if your tomcat servers can communicate with each other using Multicast with the following commands:
$ ping -t 1 -c 2 228.0.0.4
PING 228.0.0.4 (228.0.0.4): 56 data bytes
64 bytes from <server_1_ip>: icmp_seq=0 ttl=64 time=0.076 ms
64 bytes from <server_2_ip>: icmp_seq=0 ttl=64 time=0.645 ms
While everyone agrees with the value of eating your dog food, some people claim that this principle cannot be applied to all software industries.
Let’s take the Medical Health Care Industry. Who should build software for Doctors and Nurses to be used in the hospital? Its very unlikely that Doctors will start building software in the side. How do apply this principle here?
What we have today is a bunch of people trying to build software for the hospitals (most of them have not clue on how a hospital operates, those who know a little become Subject Matter Experts and take charge). Similarly there are lot of other industries.
You ask their users how they like the software and you would know. Its not that the development team did not do a good job of building the features right or the business did not do a good job of articulating what they want well. Its just that this model is setup for failure.
The Agile community realized that, they need to bring the users in and collaborate with them much more.
The Scrum community identifies one person or a group, call them Product Owner. They are part of the planning meeting, daily scrum and even the retrospectives & demos. Some (0.1%) of teams are able to get actual users during their demo. Are they confused about the PO being their User?
The XP community demands an onsite customer who can guide the team not just during planning, but also during execution. Again the same confusion exists. But the situation is slightly better.
Having said that, I really appreciate XP for pushing the knob on automated testing. Automated Testing (esp. Developer testing) is a great way to eat your own dog food. Remember how useful your API tests have proved to be. Tests are clients to your code and they consume your code by acting as client.
The Design (UX) Community are lot more User focused and tend to spend more time with the actual Users, but that’s very sporadic
The Lean community have realized that they need to have the development team sit with the business in their work area. They have realized that there are a lot of important lessons to learn from the context of the work place.
Personally I think we need to go way beyond this. If you look at some organizations (esp. Web 2.0 companies and Open Source Projects) they are their own Users. We can certainly learn something from them.
How can we do this? Here are some ideas:
At least to start with, have the team members take a formal education in the domain they are building the software. Do some case studies and then, spend quality time with the Users (actual Users). Not just interviewing them, but actually working with them (at least shadowing them or being their apprentice).
Educate the Users more about Software development process and have them work with the team for at least a week or two to under it.
May be hire people who have actually worked in the field. (You want to make sure their knowledge is up-to-date and they actually know the business really well). Also very important to maintain a good ratio. 1 member for a 10 people team is scary.
Build tools that can help the actual end users build/configure their software. As developers we build tools which we use on our own projects. Same tools (which were driven by eating their own dog food) can now be used by others to build their software. For years, creating a web presence for a company was a specialist’s job. Today with Google Apps and others, anyone can set up a website, add a bunch of forms, set up email accounts and all that Jazz. The line between a specialist’ role and a business user is blurring. Coz we have the tools to help. Esp. tools built by people for their personal use.
Again all of this can get you one step closer. But nothing like eating your own dog food.
On the ProTest project, instead of using the Election metaphor, we had also considered using the ‘Google Search Engine’ Metaphor.
Each Test is like a web page on the internet with the content we are searching for. The ProTest Search Engine would go through all the test and give a TestRank (page rank) based on various relevance algorithms (strategies). Finally our search engine would do a rank aggregation and present the tests in a prioritized order.
If we went down the Google’s search engine metaphor, we would have ended up with a highly parallel/distributed test ranking algorithms. We would have also cached the tests and associated data.
As you can see, this design would be very different from what we have now by using the Election metaphor. System Metaphors are really powerful and helpful, if used well.
On the ProTest project, we are using the Election as our System Metaphor to identify the key objects and their interactions. .i.e. to explain the logical design.
ProTest is a library to prioritize your tests such that you get fastest feedback by executing tests that are most likely to fail first. We use different strategies like a Dependency strategy which orders the tests based on dependencies of recently changed classes. If class ‘A’ was changed, then it makes sense to run all the tests that have a dependency on class ‘A’ first. We plan to have other strategies like Last failed test strategy which will order tests based on all the tests that failed in the last run, first. Others using cyclomatic complexity, test coverage and so on.
The Election Metaphor
All the tests that need to be executed are candidates standing for election (trying to get executed first). Each strategy is a voter, who votes the candidates. (We had to slightly change this metaphor. In our case, a voter can vote for multiple candidates.) Once all the voters cast their votes, we do a rank aggregation to determine the winners and hence come up with a prioritized list of tests. We plan to further enhance the metaphor to provide different weightages for each voter. Basically some voters are more powerful than the others.
Recently I was explaining the project to team from Bolivia and this metaphor really helped. I wonder if this metaphor would make sense to the Chinese.
Programming is “the action or process of writing computer programs”.
Programming by definition encompasses analysis, design , coding, testing, debugging, profiling and a whole lot of other activities. Beware Coding is NOT Programming. Depending on which school of thought you belong to, you will define the relationship and boundaries between these various activities.
For Example:
In a waterfall world, each activity is a phase and you want a clear sign-off between each phases. Also these phases are sequential by nature with very limited or no feedback. Hence you are expected to have the full design in place before you can code. Else, what do you code?
In RUP (so-called Iterative and Incremental model) even though it follows a spiral model with some feedback cycle every 3 months or so, one is expected to have the overall architecture of the project and a documented design (in UML notation) of the subset of use cases planned for the current spiral ready before the construction (coding) phase.
In the unconventional model (where we don’t have process & tool servants and team members can do what they think is most appropriate in the given context), we fail to understand these sequential, rigid processes. We have burnt our fingers way too many times trying to retrofit ourselves into this sequential, well-defined process boundaries guarded by process police. So we have given up the hope that we’ll ever be as smart as the rest of the “coding community” and have chosen a different route.
So how do we design systems then?
Some of us start with a test (not all, but just one) to understand/clarify what we are trying to build.
Some teams also start by building a paper prototype (low-fidelity prototype) of what they plan to build and jump straight to the keyboard to validate their thought process (at least once very few hours).
Yet some others use plain old index cards to model the system and start writing a test to put their thoughts in an assertive medium.
This is just the tip of the iceberg. There are a million ways people program systems. We seem to use a lot of these practices in conjunction (because they are not mutually exclusive practices and can actually be done in parallel).
People who are successful in this model have recognized that they are dealing with a complex adaptive system (CAS) and not a complicated system, where you can define rigid boundaries and be successful. In a CAS, there are multiple ways to do something and if someone makes a claim that you always have to do X before Y, we can sense the desire of putting rigid constraints which by nature are fragile. This is the same reason why there is no such thing called Best Practices in our dictionary. Instead we keep an eye on emerging patterns. If we want to see a particular pattern impact the system, we introduce attractors. But if we don’t want a pattern to impact our system we disrupt that pattern. (rip-off from Dave Snowden, creator of the Cynefin model and leading figures in Knowledge Management Community)
The open source community in general, is yet another classic example which fits into the unconventional category. I’ve never been on an open source project where we had a design phase. People live and breath evolutionary design. At best you might have a simple wiki defining some guidelines.
Anyway, I’m not saying that upfront design is bad. All I’m saying is don’t tell me that one always has to design first. In CAS, you tend to “Probe-Sense-Respond” and not “Analyze-Respond”. In software a lot of times “Action precedes Clarity”.
Today I’ve decied to leave an old friend behind. I really cannot keep up with SF’s speed any more. Its time to move on. I’m slowly going to move all my open source projects to Google Code or GITHub.
I’m a big Eclipse fan. Since 2002 I’ve been promoting Eclipse inside my company and at local conferences. In 2003, I gave a Technical Overview of Eclipse platform @ Linux Bangalore Conference (now known as FOSS.IN). My claim was Eclipse is not just an IDE, its a platform to build IDEs and other interesting applications. People laughed at me saying I was fantasizing. They claimed that Eclipse was nothing more than an IDE. Anyway, time has proved that Eclipse is really a platform that has contributed a great deal to the software world.
While I continue to be a proud Eclipse supported, over the last few days Eclipse has started disappointing me. Last whole week, Eclipse kept crashing on my machine if I upgraded some Eclipse platform plugins using their update-site. Today I stumbled upon a new issue. On my Mac, when I try to launch Eclipse.app, it complains “The Eclipse executable launcher was unable to locate its companion shared library”.
It turns out that in my Eclipse.app/Contents/MacOS/eclipse.ini file launcher.library points to a old plugin folder that does not exist. I had to manually go and update that folder to the correct folder name (version differences).
I had to change :
–launcher.library
../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.100.v20080509-1800
Wow! Today you can download CrossOver for Mac and Linux. You can also download CrossOver Games for Mac and Linux for free. This offer is only valid on 28th Oct 2008. Currently their main site is down, but you can download these softwares from http://down.codeweavers.com/
Both, Open Space Technology (OST)* and Open Source Software (OSS) are structured around volunteerism (.i.e people are the center of the universe). They really encourage people to step up and help. Volunteerism applies at many levels.
In OSS,
you can author a piece of software and put it out to help others OR
you can join an existing project and contribute to it.
you can use the software and give feedback to the authors. Filling in a bug report or requesting an enhancement is a great way to participate in this movement. There are million ways to contribute to OSS, you can write some user document for the project or give a demo to your local user group.
Similarly in OST, you can propose a topic and be a facilitator or participate in a topic proposed by someone else or blog about your thoughts.
With Volunteerism comes responsibilities and rights. You have the right to participate and contribute. If you don’t like the direction in which its going, you have the right to leave or spawn off something that is how you would like it. This very behavior leads to exploration and discovery. Hence leads to innovation. While the participants have all these rights, they are generally good citizens and are very responsible & self-disciplined. You can really see a touch of craftsmanship in their work.
In my experience both these approaches are amazing at how they can involve/engage people. In OST we say,
“Who ever comes it the right people and whatever happens is the best that can happen”.
This holds true for OSS projects as well. You can’t really force somebody to contribute. Based on my experience the core for both is passionate people.
From the outside, both OST and OSS looks very chaotic in nature. One has to act, sense and respond. When someone throws open a piece of software, they have no idea how it would evolve or even if it will interest anyone. They have to act first by putting it out there. After that based on the feedback from the community (sense), the author/s have to respond. Project which are able to do this, see a big user community around it and hence innovation. Same holds good for OST topics.
Even though we try to put some (very limited) structure around them, both OST and OSS by nature are very self-organizing, self-emerging and self-adjusting or self-correcting. They also have a self-filtering nature. Its easy to leave non-passionate and undisciplined people behind. Nor are these hero-centric systems. Also the moment we try to put too much structure or process around them, they tend to break down. They don’t necessarily fail, but evolve into something very different in nature and the kind of people around it. In OST we say,
“It starts when it starts, it ends when it ends”.
Topics in OST and features in OSS evolve over a period of time. One thing influences another and another influences some third thing. This very nature makes it very adaptive. Adaptive systems are here to stay, rigid systems will soon be extinct.
* Don’t get confused with the word Technology in OST. In this context Technology means tool. Its really a meeting/conference format.
