Cacheonix cacheonix = Cacheonix.getInstance();
      ReadWriteLock readWriteLock = cacheonix.getCluster().getReadWriteLock();
      Map map = cacheonix.getCache(mapName);
      Lock lock = readWriteLock.writeLock();
      lock.lock();
      try {
         if (!map.containsKey(key)) {
            return map.put(key, value);
         } else {
            return map.get(key);
         }
      } finally {
         lock.unlock();
      }

That approach worked but it required multiple network round trips to get things done. The latest Cacheonix release adds a distributed implementation of ConcurrentHashMap. Now you can modify distributed hash maps atomically, with minimal latency, by using the new JDK-compatible distributed ConcurrentMap API:

     Cacheonix cacheonix = Cacheonix.getInstance();
     java.util.concurrent.ConcurrentMap concurrentMap = cacheonix.getCache(mapName);
     return concurrentMap.putIfAbsent(key, value);

Distributed ConcurrentMap by Cacheonix supports all four methods of ConcurrentMap:

• V putIfAbsent(K key, V value);
• boolean remove(Object key, Object value);
• boolean replace(K key, V oldValue, V newValue);
• V replace(K key, V value);

Hope this helps.

—

Regards,

Slava Imeshev
Cacheonix Open Source Distributed Data Management Framework

—

1. Architecture for scaling Java applications to multiple servers
2. Developing scalable Java applications with open source clustered cache Cacheonix

Silicon Valley Code Camp is a new type of community event where developers learn from fellow developers. It is totally free and will occur on October 6 and 7.

See you at Code Camp 2012!

—
Regards,

Slava Imeshev
Main committer, Cacheonix

—

Sources of Performance Problems

The main sources of the performance problems in production are:

• coding errors
• internal and external bottlenecks
• garbage collection (GC)

Coding errors are places in the code where the application acts in a way that leads to significant reduction of speed such as heavy repeated calculations inside a loop. Internal bottlenecks are improper, excessive synchronization, exhausted thread pools and connection pools. External bottlenecks are serially accessed data sources such as databases, file and socket I/O.

Manifestation of Performance Problems

Performance problems in production usually manifest themselves as users complaining about the application being ‘slow’. While ‘slow’ may be a matter of perception, generally it means that the application is abnormally slow. Performance problems may seriously damage your company business and must be resolved as soon as possible. The problem resolution sequence below worked well for me for many years.

Sequence for Resolving Performance Problems

1. Make sure JVM has correct heap settings. The maximum and minimum head should be set to the same value. Example: -Xmx1000k -Xms1000k.

2. Check memory consumption on the server. Make sure that the system is not swapping. Run top or ps to check if the server as a whole and the application itself is not swapping. Increase memory on the server if it doesn’t have enough memory or move some applications out.

3. Check server CPU usage. If the server is underloaded, this is usually an indication of an internal or external bottleneck. Thread dumps, heap dumps and profiling must help to identify the bottlenecks.

4. Take and analyze thread dumps. This method is great because it doesn’t require any instrumentation, takes a minute, and often is enough to isolate the problem. During the periods of slowness, take 5-6 thread dumps. Just send kill -3 if your application is running under *nix. Separate each thread dump by a 10 second interval. Check the thread dumps for places where threads remain within the same execution stack for extended periods of time. Most of the times the problem stack traces will contain application classes. Watch for thread pools all stuck in file or network I/O. Watch for exhausted thread pools. Watch for execution threads blocked on the same objects. Fix found problems.

5. Take and analyze heap dumps. Take 2-3 heap dumps during normal operation using JMap and during the system slowing down or JVM being completely unresponsive. Analyze the heap dumps using JProfiler for objects having large number of instances. Also, compare a normal heap dump with the one taken during the slowdown. Find objects counts that grew significantly since the period of normal operation. Such objects strongly indicate a memory leak. Find and fix the problem in the code.

6. Profile while running a load test. In the QA environment, run a load test the simulates the same sequence of user actions that production users are having a problem with. Use a profiler such as JProfiler to identify CPU and allocation hot spots. Fix found problems.

7. Profile in production. Profiling adds some overhead, so the application will run slower than normally. Use a single server to profile if running a cluster. To minimize overhead, configure JProfiler properly:

• Use sampling instead of instrumentation
• Add a narrow list of packages that are interesting from a user-perspective
• Do not record allocations
• Only record CPU when there are reports from users about performance issues

8. Tune GC. Enable GC in production by using command line option –XX:+PrintGCDetails. Watch for major GCs occurring abnormally often. The main causes of frequent major GCs are the application running out of heap due to memory leaks and excessive object creation and incorrect GC parameters. Profile the application for memory leaks and excessive object creation. Fix found problems. Adjust GC parameters for better performance. Tuning GC should be the last thing to do. Check our collection of useful resources on tuning Java GC.

9. Cache hard to get data. If the performance bottleneck is found and it is a database or other serially accessed data source or a hard-to-compute data source, cache that data. Here are a few articles on how to approach caching. Or drop a message in Cacheonix support forum; we’ll try to help.

Hope this helps

Slava Imeshev

Cacheonix In-memory Data Grid

DataNuclues access platform is an open source standards-based JPA API that provides persistence and query services for  a wide set of datastores. Andy Jefferson, the project lead at DataNucleus.org, has let me know today that DataNucleus Access Platform has added support for Cacheonix. Now DataNucleus applications can scale out with ease by using Cacheonix as a distributed L2 cache.

It’s great to see growing adoption of Cacheonix by persistence frameworks and we are looking forward to further collaboration with DataNucleus.

See Also:

• Configuring Cacheonix DataNucleus L2 Cache
• DataNucleus Access Platform Home

Regards,

Slava Imeshev

Cacheonix | Reliable Clustered Java Cache

The simplest way is just to put the list to the cache. Note that if you want to use a strongly typed cache, you need to provide a concrete list type:

import java.util.ArrayList;
import cacheonix.Cacheonix;
import cacheonix.cache.Cache;

...

   /**
    * Puts a list to the cache.
    *
    * @param key  a key identifying the list.
    * @param list the list to cache.
    */
   public void putListToCache(String key, ArrayList<String> list) {

      final Cache<String, ArrayList<String>> listCache = Cacheonix.getInstance().getCache("listCache");
      listCache.put(key, list);
   }


   /**
    * Gets a list from the cache.
    *
    * @param key a key identifying the list.
    * @return the cached list or null if the list wasn't found in the cache or if the list is null.
    */
   public ArrayList<String> getListFromCache(String key) {

      final Cache<String, ArrayList<String>> listCache = Cacheonix.getInstance().getCache("listCache");
      final ArrayList<String> list = listCache.get(key);
      return list;
   }

While the above works, there is a better way, which is creating a first-class object for the cached list.

Usually lists don’t come alone and are accompanied by some metadata. For example, if you are caching a database result set, in future you might want to cache time it took to retrieve the result set in addition to the result set itself. Wrapping the list into a class provides an ability to maintain a stable signature of the cached object. If you need to add more fields in the future, you don’t have to change all places where a cached list is put to and retrieved from the cache.

A cached result set object that contains a list would then look like the following. Note that it declares a signature interface Immutable that signals the cache that this object can be passed by reference:

import java.io.Externalizable
import cacheonix.cache.Immutable;

...

  public class CachedResultSet implements Externalizable, Immutable {

    private List<String> list = null;


    public CachedResultSet(List<String> list) {
      this.list = new ArrayList<String>(list);
    }


    /**
     * Required by Externalizable.
     */
    public CachedResultSet() {
    }

    public List getList() {
      return new ArrayList(list);
    }

    public void writeExternal(ObjectOutput objectOutput) throws IOException {

      if (list == null) {

        objectOutput.writeBoolean(true);
      } else {

        objectOutput.writeInt(list.size());
        for (int i = 0; i < list.size(); i++) {

          objectOutput.writeUTF(list.get(i));
        }
      }
    }


    public void readExternal(ObjectInput objectInput) throws IOException, ClassNotFoundException {

      if (!objectInput.readBoolean()) {

        int size = objectInput.readInt();
        list = new ArrayList<String>(size);
        for (int i = 0; i < size; i++) {

          list.add(objectInput.readUTF());
        }
      }
    }
  }

Getting the cached result set would look like this:

   public CachedResultSet getListFromCache(String key) {

      final Cache<String, CachedResultSet> listCache = Cacheonix.getInstance().getCache("listCache");
      final CachedResultSet list = listCache.get(key);
      return list;
   }

Regards,

Slava Imeshev

Cacheonix | Reliable Clustered Java Cache

Problems With Default MyBatis Caching

There are a few problems with the way caching works in MyBatis:

• MyBatis keeps results of all selects in a single, namespace-wide, cache. This means any invalidation caused by inserts or updates will flush cached results of all selects, including those that are really not affected by the changes.
• It is impossible to ignore invalidation completely and to have pure time-only expiration for some selects.
• It is impossible to have different cache configurations for particular selects.

Benefits of Cacheonix Adapter for MyBatis

Cacheonix have addressed these concerns by adding the following features:

1. Per-select MyBatis caches
2. Ability to turn off invalidation
3. Ability to turn off namespace cache
4. Cache templates for select caches

For more information and examples check Cacheonix documentation: Configuring MyBatis Cache Adapter.

Enjoy!

Regards,

Slava Imeshev
Cacheonix | Reliable Distributed Java Cache

Yesterday, during a meetup at Loopt, a question of an API for testing private methods was brought up. I used to use JUnit add-on PrivateAccessor [1] when I was testing private methods. The problem was that applying PrivateAccessor produced prohibitively verbose code, so you had to think really hard to understand what the test was doing.

invoke() with varargs

Luckily, Java 5 provides vargargs API that allows to write really concise code for invoking private methods and return results. Here is the invoke() method that makes use of vargargs. Feel free to use it in your code:

   /**
    * Invokes a private method on an object.
    *
    * @param obj        the object on what to invoke the method.
    * @param methodName the name of the method to invoke.
    * @param parameters a list of parameter
    * @return the result of dispatching the method represented by the object.
    */
   public static <T> T invoke(Object obj, String methodName, Object... parameters) {

      try {

         // Create a list of parameter classes
         final Class[] parameterTypes = new Class[parameters.length];
         for (int i = 0; i < parameters.length; i++) {
            parameterTypes[i] = parameters[i].getClass();
         }

         // Get method
         Class clazz = obj.getClass();
         Method method = clazz.getDeclaredMethod(methodName, parameterTypes);
         method.setAccessible(true);

         // Call method
         return (T)method.invoke(obj, parameters);
      } catch (RuntimeException e) {

         throw e;
      } catch (Exception e) {

         throw new IllegalArgumentException(e);
      }
   }

Complete Example

Below is a complete JUnit tests that tests StringBuilder’s private method append(). Notice how neatly the invocation of append() fits into a short single line:

import java.lang.reflect.Method;
import junit.framework.TestCase;

public final class StringBuilderTest extends TestCase {


   /**
    * Object under test
    */
   private StringBuilder object;


   /**
    * Tests String's private method 'append(StringBuilder sb)'.
    */
   public void testAppend() {
      
      StringBuilder toAppend = new StringBuilder("Test");
      StringBuilder result = invoke(object, "append", toAppend);
      assertEquals("Test", result.toString());
   }


   /**
    * Invokes a private method on an object.
    *
    * @param obj        the object on what to invoke the method.
    * @param methodName the name of the method to invoke.
    * @param parameters a list of parameter
    * @return the result of dispatching the method represented by the object.
    */
   public static <T> T invoke(Object obj, String methodName, Object... parameters) {

      try {

         // Create a list of parameter classes
         final Class[] parameterTypes = new Class[parameters.length];
         for (int i = 0; i < parameters.length; i++) {
            parameterTypes[i] = parameters[i].getClass();
         }

         // Get method
         Class clazz = obj.getClass();
         Method method = clazz.getDeclaredMethod(methodName, parameterTypes);
         method.setAccessible(true);

         // Call method
         return (T)method.invoke(obj, parameters);
      } catch (RuntimeException e) {

         throw e;
      } catch (Exception e) {

         throw new IllegalArgumentException(e);
      }
   }


   public void setUp() throws Exception {

      super.setUp();

      object = new StringBuilder(10);
   }
}

Making invoke() reusable

To make invoke() reusable, I suggest to move it to a separate utility class or at least to create a super class for all your tests that extends junit.TestCase and move invoke() there.

Regards,

Slava Imeshev
Cacheonix: Reliable Distributed Java Cache

References

1. JUnit Add-On PrivateAccessor

Distributed caching imposes an additional requirement towards unit tests for cache keys and cached values. Particularly, you must ensure that the object that was received at another end is the object that was sent. Here is how to tests it:

First, serialize an object to a byte array and then deserialize the byte array back to an object

      // Serialize the object
      ByteArrayOutputStream baos = new ByteArrayOutputStream(100);
      ObjectOutputStream oos = new ObjectOutputStream(baos);
      oos.writeObject(originalInvoiceKey);
      oos.close();

      // Deserialize the object in serialized form
      final byte[] serializedInvoiceKey = baos.toByteArray();
      ByteArrayInputStream bais = new ByteArrayInputStream(serializedInvoiceKey);
      ObjectInputStream ois = new ObjectInputStream(bais);
      final InvoiceKey deserializedInvoiceKey = (InvoiceKey) ois.readObject();
      ois.close();

Second, assert that the deserialized object and the original object are equal:

      assertEquals(originalInvoiceKey, deserializedInvoiceKey);

While you have already implemented methods equals() and hashCode() for the key, cached values need to have that methods added. For value objects, equals() and hashCode() need to satisfy a requirement for ‘literal equality’. In other words, to test that a value object can be serialized and deserialized, its equals() and hashCode() should include all non-transient fields:

public final class Invoice implements Externalizable {

   private int invoiceID;
   private Date invoiceDate = null;
   private int invoiceNumber;
   private int customerID;

...

   public boolean equals(final Object o) {
      if (this == o) return true;
      if (o == null || getClass() != o.getClass()) return false;
      final Invoice invoice = (Invoice) o;
      if (invoiceID != invoice.invoiceID) return false;
      if (invoiceNumber != invoice.invoiceNumber) return false;
      if (customerID != invoice.customerID) return false;
      if (invoiceDate != null ? !invoiceDate.equals(invoice.invoiceDate) : invoice.invoiceDate != null) return false;
      return true;
   }

   public int hashCode() {
      int result = invoiceID;
      result = 31 * result + (invoiceDate != null ? invoiceDate.hashCode() : 0);
      result = 31 * result + invoiceNumber;
      result = 31 * result + customerID;
      return result;
   }
}

If adding equals() and hashCode() to the cached value is impossible, use explicit assertEquals() for object’s fields:

      assertEquals(invoiceID, deserializedInvoiceKey.getInvoiceID());

Regards,

Slava Imeshev
Cacheonix | Reliable Clustered Java Cache

I am using Mac Pro and MacBook Pro as development machines while working on Cacheonix. The Mac Pro is tucked under my desk and is practically invisible. Yet, the MacBook has been occupying a noticeable chunk of the desk surface, which is not nice:

The obvious solution was to set it upright, but how? My requirements to the solution were: simple; stable; good-looking.  An extensive research fetched a winner – Power Support’s Docking Stand for MacBook. This neat device satisfies all my requirements. The assembly includes pads to prevent scratching. The aluminum color that matches my MacBook’s body is a nice bonus. $40 price tag is a steal.

Assembly has taken 2 minutes and after getting it all together the MacBook has practically disappeared from the surface of the desk freeing about 50 sq. in:

Too bad PowerSupport stopped making those.  Henge Docks now makes a more advanced and a bit more expensive model that allows to really dock your Macbook.

Regards,

Slava Imeshev
Cacheonix: Reliable Distributed Java Cache