Images: Managed Images

You'd think the friendly folks at Sun might mention this... but it's not really talked about it. In fact, it's so not talked about that I still don't have a firm understanding of what it is. Once I googled it and found a nifty little diagram explaining the idea, but I've never seen that diagram again.

Here's my understanding of the concept:

managed image: this is a BufferedImage that is optimized to run efficiently in memory.

Of course by default your images are managed. But there are 2 seemingly innocent ways to unmanage them. Once an image becomes unmanaged: there is no way to bring it back to its managed state. They are blacklisted. They are on the terrorist watchlist of your JVM. Rendering to an unmanaged image can be a factor of over 100x slower than rendering to a managed image.

The two ways I know to unmanage an image are:

  1. Access the raster's DataBuffer. For example:
    myBufferedImage.getRaster().getDataBuffer();


  2. Add an ImageConsumer to a BufferedImage's ImageProducer. For example:
    myBufferedImage.getSource().addImageConsumer(c);

Now that I've scared you into listening, here's the great news: this problem has been mostly taken care of in recent years. Based on my own experience and the comments in this article: this only appears to be a problem if you're using a Mac with Quartz rendering.

To study this further yourself, I recommend running this sample program.

Here is sample output for that program from a few years ago, running Mac 10.4.9 PPC and Java 1.4:

Regular Test Time (1): 6
Unmanaged Test Time (1): 11494
Regular Test Time (2): 5
Unmanaged Test Time (2): 11384
Unmanaged Test time (2): 10129

So regardless of how I unmanaged my image, the results are about the same: my performance change to about 2000 times slower than it should be.

Also note that if you remove the ImageConsumer it doesn't matter: the damage has already been done.

In a way this makes sense: if I have the DataBuffer in hand, then I may be hijacking the image contents. So Java has to constantly check to make sure that the DataBuffer and the Graphics2D match up. Also if I have an ImageConsumer listening to the ImageProducer: this adds a level of complexity to how the image data is manipulated.

Here are tips for avoiding any problems. Even though the threat is largely passed, I still follow these practices out of habit/safety:

  • If you never need to interact with the Graphics2D of an image, then you shouldn't see a performance problem. That is: suppose you just wanted to define an image pixel-by-pixel, but you never wanted to use a Graphics2D to manipulate that image.

  • Suppose you do want to control the pixels in an image? Not a problem: use bi.getRaster().getDataElements() and bi.getRaster().setDataElements(). Just don't ask for the DataBuffer.

As a footnote, Jerry recently pointed out that calling myBufferedImage.setRGB() used to unmanage an image. This does not appear to be the case in Java 1.4 or later.

16 comments:

  1. "...myBufferedImage.setRGB() used to unmanage an image. This does not appear to be the case in Java 1.4 or later."

    If I recall correctly, prior to Java 1.4 there was no such thing as a managed image. I think managed images appeared on the scene around the same time as Volatile Images. If not a little later. Managed images make most uses of VolatileImage obsolete.

    ReplyDelete
  2. Interesting you'd say that. I remember trying on VolatileImages on Mac and not seeing an improvement. Maybe that's why.

    ReplyDelete
  3. I just ran ManagedImageTest under Java 1.5.0_14 and XP and got the following results:

    Regular Test Time (1): 15
    Unmanaged Test Time (1): 0
    Regular Test Time (2): 0
    Unmanaged Test Time (2): 15
    Unmanaged Test Time (2): 0

    This corresponds to my experiences of using getDataBuffer() and Graphics2D in parallel. Seems that the problem does not appear anymore (at least) up from Java 1.5(?)

    ReplyDelete
  4. Hurray! You appear correct. I was worried your results might reflect a difference in XP vs Mac, but these are my latest results on my Mac 10.5.4 laptop:

    Running Java 1.4:
    Regular Test Time (1): 6
    Unmanaged Test Time (1): 2597
    Regular Test Time (2): 6
    Unmanaged Test Time (2): 2563
    Unmanaged Test Time (2): 2580

    Running Java 1.5:
    Regular Test Time (1): 7
    Unmanaged Test Time (1): 7
    Regular Test Time (2): 7
    Unmanaged Test Time (2): 7
    Unmanaged Test Time (2): 7

    Running Java 1.6:
    Regular Test Time (1): 7
    Unmanaged Test Time (1): 3
    Regular Test Time (2): 3
    Unmanaged Test Time (2): 4
    Unmanaged Test Time (2): 3

    ReplyDelete
  5. There seem to be some problems I have come across with setRGB() on a buffered image. Suffice to say, using Java1.6 and a rather large data set (over 13,000,000 pixels), the processing (in Windows) takes 359 seconds (approx), compared to 4 seconds for the same data set in Linux (my development platform). I am still trying to work out why such a discrepancy exists.

    ReplyDelete
  6. Hmm. That could be related to several things. (First of all: what's the relative processor speed on your Windows vs Linux machines?)

    Also on Windows the pixel format of choice is BGR, I believe. What is it on Linux? If it's RGB, then Windows may have a LOT more work to do: it has to flip every RGB int into a BGR int, which will be painful.

    Or it makes a difference which setRGB() method you're referring to: one pixel at a time or a block at a time? I wonder if you'll see improvement more directly calling:
    image.getRaster().setDataElements()

    So those are some first reactions to the issue; let me know if you find anything out.

    ReplyDelete
  7. Thanks for your reply...

    The machines are like-for-like - in fact, the same machine running under dual boot.

    I have tried the two versions of setRGB, as well as your suggested image.getRaster().setDataElements() where the colour information is held in an int array. They all require very similar (342 seconds instead of 359 seconds on the Windows platform, 2 seconds instead of 3 or 4 on Linux) loading times.

    To make the int array, I have had to cycle through all the pixels, which may in fact be the major bottleneck, but as I have three BufferedImages, each containing a colour channel (loaded in from separate files), I am not sure how to avoid this. It is of the form:
    redC = redBuff.getRGB(x,y)
    greenC = greenBuff.getRGB(x,y)
    blueC = blueBuff.getRGB(x,y)

    These are then combined into a single int for colour, and now added to the rgbArray, to be set using theimage.getRaster().setDataElements() method.

    ReplyDelete
  8. Further to my previous post, I have managed to speed the entire process up (sub 1second on Win & Lin) by using redArray = (byte[])redBuff[i].getRaster().getDataElements(0, 0, redBuff[i].getWidth(), redBuff[i].getHeight(), redArray);

    // the same for green and blue

    The fact that it is a byte array does mess with it a bit though, as it seems to miss out any pixels whose intensity is not high enough, so I am losing data.

    Thanks for this page though, as it has been a help, and pointed me in the right direction.

    ReplyDelete
  9. If you'd like me to keep commenting on this subject, how about you email me directly with source code? Back-and-forth exchanges in blog comments will get tiresome. :) See my blog profile for email info.

    ReplyDelete
  10. Are you ready for the red pill?

    System.setProperty("sun.java2d.allowrastersteal", "true");

    (I haven't tested it myself. Might not work anymore.)

    Read here and here (and here for more interesting flags).

    The Sun'er Dmitri Trembovetski is really helpful on several forums, e.g. javagaming.org and
    forums.java.net/java2d and forums.sun.com/java2d.

    ReplyDelete
  11. That is really interesting. :) But if it was magical and bugproof: the property would be active in the JVMs (where it applied) by default. So it is good to know, and may be very useful in some applications/instances, but I would be reluctant to consider it universally "safe" unless it were better documented. Meanwhile JavaFX is supposed to use hardware accelerated graphics whenever possible, so that's promising.

    ReplyDelete
  12. I believe the hardware acceleration came with java 1.6 update 10 - with this release, all java2d and swing stuff is done using Direct3D.

    That allowrastersteal flag have apparently been around for a long time, and my best guess is that it'll stay there a while. It is a good feature - one that I actually find a bit strange isn't available by proper API calls. It is a kinda "grown-up" thing: To use it, one need to understand that there really are two copies of ones image: One that is "pixled" in actual int-arrays, and one that resides on the graphics hardware - and that transferring to and fro that is a costly operation.

    ReplyDelete
  13. Seems this is obsolete. Under 10.6.x:

    Regular Test Time (1): 6
    Unmanaged Test Time (1): 3
    Regular Test Time (2): 3
    Unmanaged Test Time (2): 3
    Unmanaged Test Time (2): 3

    ReplyDelete
  14. My results under OS X 10.5.8 and the default Java shipped (1.5) are different, it seems you still need the managed stuff:

    $ java ManagedImageTest
    Regular Test Time (1): 4
    Unmanaged Test Time (1): 1825
    Regular Test Time (2): 4
    Unmanaged Test Time (2): 1808
    Unmanaged Test Time (2): 1826

    $ java -version
    java version "1.5.0_20"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_20-b02-315)
    Java HotSpot(TM) Client VM (build 1.5.0_20-141, mixed mode, sharing)

    Any idea why my results seem to be different than your tests for 1.5?

    ReplyDelete
  15. A., I can confirm what you're seeing.

    Earlier (Oct of 2008) I commented that Java 1.5 was performing well. But just now I saw times similar to yours.

    My theory is it's related to Quartz. In the summer of 2009 Apple quietly changed the default graphics pipeline: we used to use Sun's, and then they changed it back to Quartz.

    I toggled the "apple.awt.graphics.UseQuartz" property, and that seemed to be the key (if it's "false", then I get great performance again). This makes yet another reason I'd recommend turning off Quartz. :)

    Glad you noticed; I had assumed the problem had gone away. Turns out its still lurking.

    ReplyDelete
  16. Jeremy, thanks for the hint. If I disable Quartz (java -Dapple.awt.graphics.UseQuartz=false ManagedImageTest) the managed/unmanaged test times are the same. Interesting. I would assume that the Quartz backend is at least on par with the Sun 2D backend (and can potentially be faster if it can utilize hw acceleration) but apparently it's not the case. Maybe it's no problem from 10.6 (which ships with Java 1.6 I believe) onward.

    ReplyDelete