Layer Masking Effect - Technical Blog

Layers masking effect using OpenGL ES shader with libGDX

In this article I will explain the basics of using texture alpha masks with multiple layers and how to create complex effects using this neat trick. The source code will be based on the libGDX framework and written in Java. Recently I’ve used this code for intro and menu effect in my last Android game. Custom pixel shader used with SpriteBatch will be required for this effect to work. Honestly I don’t know if I’m using the correct name, but basically what this effect does is that it takes as input two textures, the first one is called the layer texture and second the mask texture. The mask texture determines the opacity of each pixel. The additional enhancement is to apply rotation, scaling and translation to those layers to create complex effects like in the three examples below:

Layer Masking Effect - Hexagon Circles Effect Example - Technical Blog Layer Masking Effect - Biohazard Lines Effect Example - Technical Blog
Layer Masking Effect - Retro Futurism Effect Animation - Technical Blog

What is libGDX?

Libgdx is a Java game development framework which provides an unified cross-platform API. LibGDX allows to easily deploy OpenGL ES applications to desktop (because of JVM), Android and iOS (using Intel Multi-OS Engine). It supports also 2D physics (there’s a JNI wrapper for Box2D) and 3D physics (JNI wrapper for Bullet Physics). The bitmap fonts can be generated on the fly from TTF files which is really useful (plus there’s support for internationalization and localization). I won’t explain here how to setup a basic libGDX project, more information can be found here. LibGDX is a framework not a complete game engine, so in order to have a complete game some fair amount of coding is required. However using Intellij IDEA or Android Studio is quite a pleasant experience, there’s a lot of examples and the API is really well written.

Layer information structure

Each layer of the effect has some unique properties like position, size, current rotation, used textures, scaling and so on. One single effect can be composed of multiple layers. Those layers will be later rendered using custom shader setup and SpriteBatch (2D rendering). Note that because second texture unit is being used with this shader the sprite batch needs to be flushed after every layer (here flushing means that any pending sprites are rendered and the batch becomes empty again). Let me first explain the basic structure of LayerInfo class:

• Texture texture layer standard texture (diffuse, texture unit index 0)
• Texture maskTexture additional mask texture (texture unit index 1)
• Vector2 size size of the layer (and the mask) in pixels
• Vector2 position current position (center)
• Vector2 realPos real position (rectangle corner)
• Vector2 offset current mask offset value in screen pixels
• Vector2 maskOffset current offset for mask in UV space
• Vector2 layerOffset current offset value in screen pixels – layer only
• float speed speed of the effect (when scaling/pulsing)
• float direction current direction of the effect (grow / shrink)
• float rotation current rotation of the layer (centered) in degrees
• float rotationSpeed rotation speed (degrees per second – can be less than < 0.0)
• float scale current scale value
• float maskScale current scale value of the mask
• float layerScale current scale value for layer only
• float minScale minimal scale allowed
• float maxScale maximal scale allowed
• Interpolation interpolation interpolation method to use when calculating scale (sine as default)
• float scaleRatio ratio value (between 0.0 and 1.0) to use with interpolation function
• Mode scaleMode what element should be change with pulsing (scaling), layer only, mask only or both?
• Mode offsetMode for which texture should the offset be applied to?

Applying offset and scaling to layer

Offsets and scaling can be applied separately or together to both layer and masking texture. That’s why scaleMode and offsetMode are used. If offsetMode is set to ‘layer only’ then the final position of the layer 2D rectangle will be moved by number of pixels set in layerOffset vector. If it’s set to ‘mask only’ then the offset is converted to UV space and the special shader uniform variable will be updated. The same method applies to scaling (there is a separate scale for the layer and the mask). The offset needs to be set externally – the scale is interpolated in LayerInfo.update() function. The ‘delta’ parameter is in seconds.

Layer position is stored separately in ‘position’ and ‘realPos’ – one describes position aligned to bottom left corner and the latter describes the final centered position of the layer. The scaling and rotation needs to be aligned to center, also the SpriteBatch.draw() takes quite complicated list of parameters to draw a rectangle that is scaled, rotated and aligned. The rotation angle is in degrees.

Offset needs to be set by externally using special functions. Offset value depends on what type of scaleMode and offsetMode is being used and on current scale value. Value of mask texture coordinate offset needs to be scaled properly to achieve more natural effect (otherwise it would move too quickly and the layer position would not align with the mask texture). To achieve the effect in which the texture layer is still and the mask texture is moving over it revealing different parts of the original texture, scaling needs to be turned off (‘shouldPulse’ set to false) and offsetMode needs to be set to ‘both’. Why offset needs to be applied both to layer (as translation in screen space) and to UV coordinates? Position of the rectangle on which both textures are being drawn is not static, so texture coordinates need to change accordingly to compensate for this movement. More examples of this in later parts of the article.

Drawing single layer

Full source code for LayerMaskingEffectDrawer and LayerMaskingEffect will be presented at the end of the article. The LayerMaskingEffect contains an array of LayerInfo‘s and a pointer to external AssetManager (provided by libGDX API) used for easy retrieval of texture objects. Also the class for layer masking effect contains helper functions for drawing, adding new layers and adjusting offsets for all layers in the effect. Remember that custom shader code (pixel shader) is being used with the sprite batch (which is initialized in LayerMaskingEffectDrawer). Various shader uniforms need to be set before drawing. This includes single floats for the mask scale and whether or not the mask texture is being used. The mask texture needs to be bound manually. After drawing the textured layer rectangle the sprite batch needs to be flushed. It’s because of the second texture unit. Without flushing the batch, all layers would be drawn using the same mask texture (whichever was bound as last). Below is the code used for drawing layer directly onto the screen.

Shader source code

Various examples

Layer Masking Effect - Triangles Animation Example - Techical BlogEver seen wallpapers with triangles and space photos?
This effect is quite easy to achieve – better use 8 different layers configured separately than to use one mask texture containing 8 triangles. That way it will be more flexible. Also, each triangle can have different scaling/offset method and different layer texture. Here each layer has fixed position and three of them are rotated by 180 degrees. Offset is set externally – there is no interpolation method for layer/mask offsets in LayerInfo.update() function. Scaling is disabled and offsetMode is set to ‘mask only’.

Layer Masking Effect - Hexagon Circles Effect Example - Technical Blog

In this example the effect is using 4 layers. Layer offset is not being changed – just the scale factor is updated. For each layer the scale is applied to both texture coordinates and the whole layer itself – texture coordinates need to change accordingly to compensate for larger size of the layer. In the end this looks like those shapes (created by the mask textures) are revealing what is underneath them. Many parameters can be fine-tuned – like the minimal and maximal scale and scaling speed.

Complete source code


Layers Masking Effect (git repository)
Layers Masking Effect (git repository)
33.9 MiB
Layers Masking Effect (no git repository)
Layers Masking Effect (no git repository)
16.4 MiB