Object Pick Selection Using Framebuffer and Picking Box - flexigame.com Technical Blog

Object Pick Selection Using Framebuffer and Picking Box


Pick selection (or mouse picking) is a way of selecting objects in 3D scene. It can be very useful for strategy games or any kind of 3D editors. There are different methods to achieve this. Most of them require ray casting (projecting 3D ray from mouse position, through the camera and into the scene) and converting mouse click position from viewport space to worldspace position. With ray origin position and direction vector it is possible to compute intersections with various primitive shapes (like spheres, AABB, OBB, triangles). In this article I will describe in depth how to setup pick selection class with multiple options and methods. Pick selection in this demo will use ray intersection with a sphere, AABB triangles, FBO color buffer and selection box. The full demo and source code is written in Java using libGDX and Intellij IDEA. I’m using libGDX for my first demos/tutorials because it’s much faster to write prototypes this way. Source code for pick selection can be quite easily converted to C++. Please note that mathematical subject (geometric intersection testing) won’t be fully explained in this article – main focus is on OpenGL and programming solution in general. I’m also assuming that anyone reading this article has some experience with programming, OpenGL graphics and libGDX. At the bottom there is a download link for full source code and project files, .jar package is also available for download.


Ray casting in libGDX

Fortunately Camera class supplied with libGDX provides easy methods for getting picking ray parameters (origin and direction vectors). There are also methods for projecting given world space coordinates to screen coordinates (projecting point in 3D space onto the final 2D screen). This truly simplifies things.

Unprojecting is done a little bit differently – it won’t be that accurate, screen coordinates will be translated into world space where z-coordinate of 1.0 will return a point on the far plane of the camera frustum and a z-coordinate of 0.0 will return a point on the near plane.

If someone wants to convert the source code presented in this demo to C++, it will be quite easy to use OpenGL Mathematics library (https://glm.g-truc.net/) which supplies functions for complex matrix transformations including functions like project and unproject.


Different methods for pick selection

Picking ray intersection with sphere

This is one of the easiest methods for selecting objects in a 3D scene. It requires central position of each object and radius of the bounding sphere. This method won’t be very accurate if some of the objects are large and have irregular shape. Furthermore, bounding spheres can overlap on each other on the screen which will cause multiple or wrong selections. This can be avoided by additional checks for objects that are closer to the camera (although not implemented in this demo).

Picking ray intersection with bounding spheres
Picking ray intersection with bounding spheres

Picking ray intersection with Axis-Aligned Bounding Box

This method can be sometimes even less accurate when compared to intersection with spheres. LibGDX provides ready to use static functions in class Intersector for checking whether or not the picking ray is intersecting with axis-aligned bounding box.

Ray intersection with axis-aligned bounding boxes
Ray intersection with axis-aligned bounding boxes

Picking ray intersection with transformed AABB triangles (Oriented Bounding Box triangles)

This method is more accurate then the previous two, however it requires more processing power. More operations need to performed not only for transforming all 8 corner points of model’s bounding box, but also for checking ray intersections with each one of 12 triangles that make the oriented bounding box of the current object model. Because of the way the bounding sphere radius is being calculated, it is certain that OBB will be always completely inside the sphere.

Ray intersection with OBB triangles
Ray intersection with OBB triangles

Picking ray intersection with triangles of the convex mesh

Another way for selecting 3D objects is to use separate convex meshes for each object. These would need to be prepared and loaded externally. This method requires additional work to prepare convex meshes in 3D modeling software. The advantage is that such meshes could be used also with occlusion culling procedures (if correctly prepared). This way is not implemented here.

Mouse pointer intersection with the on-screen rectangle

This method similarly to checking OBBs requires transformation all 8 corner points of model’s bounding box. Additionally each point after multiplication by transformation matrix  needs to be projected onto the screen (converting world space coordinates to screen space coordinates). Using on-screen rectangles for each object makes more sense when using it with picking/selection box.

Mouse pointer intersection with on-screen boxes
Mouse pointer intersection with on-screen boxes

Using framebuffer and checking pixel data of the color texture

This method is the most accurate – it uses additional framebuffer object and texture. Additional rendering is done using custom fragment shader code – the scene needs to be rendered twice. Next, the pixel data needs to be read to CPU side buffer – this can also be optimized by loading only pixels covered by the selection box (or just one pixel for comparison if selection is done by a single point click). Further optimization can be done by checking pixels covered only by the intersection of the main selection box with the on-screen rectangle of the given object.

Drawing intersections of the picking box with the object’s on-screen rectangles
Drawing intersections of the picking box with the object’s on-screen rectangles
Another example of selecting objects based on the framebuffer’s pixel data
Another example of selecting objects based on the framebuffer’s pixel data

Definition of the SpatialObject interface

The SpatialObject interface will be used with the scene manager and the pick selection class. It provides functions for getting and setting spatial parameters like scale, transform, radius, position and so on. In order to work properly, the pick selection class needs the position, radius and bounding box dimensions of every visible object in the scene. The full list of spatial objects will be traversed when needed and each object will be queried.


Overview of the SimpleSceneManager class

Screenshot SceneManager Overview
Scene manager class used in this demo provides just basic functions that are needed in 3D application. Hierarchy of scene nodes in 3D space is not used in this example (like octree or BVH). Just linear traversal of the game objects will be sufficient, as the number of all objects will be quite low.  Additional class GameObject extending ModelInstance and implementing SpatialObject interface is used. A game object class also has additional member variables for bounding box, object name, scale and so on. The scene manager also provide functionality to draw debug shapes for each game object using the ShapeRenderer, like AABBs and bounding spheres.

Rendering game object's debug shapes (bounding spheres and boxes)
Rendering game object’s debug shapes (bounding spheres and boxes)

Frustum culling is implemented in an easiest possible way and is required for the pick selection to work properly. LibGDX Camera class already provides functions for checking whether or not primitive shapes are inside of the viewing frustum. Valid position and bounding sphere radius of each object is required.

AbstractFlags class definiton

The scene manager class also uses special AbstractFlags class for enums and on/off options. It is much easier to use such flags in similar way like in C++ and it doesn’t require to have separate boolean variables for each option.

Abstract flags used with the scene manager are extended to contain options for frustum culling and drawing debug shapes.

GameObject class definition

The game object class extends directly ModelInstance class – this way it can be used directly with ModelBatch for quick and easy rendering. GameObject class also contains additional variables for storing original bounding box (aabb of the model), center position, extent vector, radius, scale and other useful data. All these are needed for frustum culling and pick selection to work. Additionally the GameObject implements the SpatialObject interface so it can be easily used with the PickSelection traverse functions.

Game objects bounding box and sphere radius needs to be kept up to date and store valid data even when scaling is applied to the transformation matrix.


Customizing CameraInputController

LibGDX also provides the class for easy manipulation of camera position and orientation, however the default class does not have the most intuitive controls. In this demo the camera controller is implemented by extending the CameraInputController class. WSAD keys are used for translating camera origin position, left control moves camera downwards and spacebar moves camera upwards. The right mouse button is used for rotation (hold the right mouse button and then move the mouse around to rotate the camera).


Overview of the PickSelection class

List of on/off options for pick selection

AbstractFlags class is extended inside of PickSelection class to provide various on/off options for customizing class behavior and for saving inner state values. Most of those options change the way the list of spatial objects is being traversed when needed and how each object is checked for ray intersections.

  • SELECTION_ON_CLICK – selection works only when picker is active,
  • SELECTION_ON_HOVER – selections is active even without toggled click,
  • CHECK_AABBS – whether or not to check axis-aligned bounding boxes of each spatial object,
  • CHECK_ON_SCREEN_BOXES – whether or not to test on-screen boxes; the spatial object’s AABB is transformed and projected onto the screen,
  • CHECK_OBB_TRIANGLES – whether or not to check ray intersections with triangles of an OBB – sometimes may be more accurate than checking bounding spheres (default),
  • CHECK_FBO_PIXELS – whether or not to check the frame buffer pixel data (which is provided externally),
  • PICKER_ACTIVE – whether or not the picker is active (is selection activated?),
  • GROUP_SELECTION_MODE – used for activating multi-selection mode,
  • TOGGLE_SELECTION_MODE – special mode where the second click (second selection) deselects the spatial object,
  • USE_PICKING_BOX – whether or not to use the on-screen picking box for selecting multiple objects,
  • INTERNAL_BEGIN – internal flag marking begin() function execution – set to false on end()
  • INTERNAL_SHOULD_UNSELECT – internal flag for force removing selected objects (when picking ray does not collide),
  • INTERNAL_SHOULD_CONTINUE – internal flag telling whether or not the traversal should continue.

Of course, some of those options exclude each other (like for example SELECTION_ON_CLICK and SELECTION_ON_HOVER), and others can be active simultaneously.

List of main member variables

• Array<SpatialObject> spatialObjects external array with spatial objects (that need checking)
• Array<SpatialObject> selectedObjects array with currently selected spatial objects
• ObjectMap<Integer, PickingInfo> pickingInfoMap special map for mapping object scene index to PickingInfo object
• StateFlags stateFlags current state flags (on/off options)
• Camera camera external camera (from the scene manager, needed for ray origin and direction)
• Ray ray current picking ray info
• Vector2i pickPos currently reported pick selection position
• Vector2i pickPosBegin start position of the pick selection (where picker was reported as active)
• float pickTimeStampBegin picking beginning time stamp
• Rectangle pickBox selection box rectangle
• Result goodPickResult what is considered a good pick result
• long initTimeStamp time stamp in milliseconds marking PickSelection object initialization
• Vector3[] aabbPoints Vector3 array for eight corner points of the bounding box
• BoundingBox internalAABB internal bounding box – helper variable used for picking box additional detection
• Vector3 tmpVec temporary helper vector
• PixelChecker fboPixelChecker external interface for checking fbo pixels (for objects ids)

Definition of the PixelChecker interface

This is a special interface used for checking pixels of the color texture that was rendered off-screen using the framebuffer. Each color value of the pixel (RGBA) can be converted to single 32 bit integer value that represents index of the rendered object (if any). This will help determine which object is under the mouse pointer. Such framebuffer and rendering procedures need to be handled outside of the PickSelection class and traverse methods, but in exchange this interface is provided. If available it is used alongside the code that checks for ray intersections.

Inner PickingInfo class definition

This is a helper class used for storing extended information about given spatial object. This information includes intersection point (if any), the dimensions of the on-screen rectangle (not always available), picking result, timestamp and others. The class implements the Poolable interface for better memory management (meaning that new objects and taken from the object pool if available).

Definition of the OnSelectionListener interface

There’s an easy way to provide multiple listeners for handling object selections on the fly. Those listener won’t always be called in bulk (when deselecting all objects at once).

internal_isPicked(PickingInfo) function definition

In this function ray intersections are checked with chosen spatial object and the PickingInfo object is updated depending on the tests results. Here is also where the point transformations are performed and bounding box’s corner points are projected onto the screen, so this function can be quite time consuming (and it should not be called multiple times in a row for the same spatial object). The test result is later used in performFullCheck(…) function to finally determine whether or not given spatial object is selected (which also depends on current selection mode and not only on ray intersection).

performFullCheck(SpatialObject) function definition

This function is called from the main traverse() function. It determines whether or not to mark given spatial object as selected and add it to the list.

begin() function definition

This function is used for setting internal flags to proper values. Depending on the list of active options, different internal flags are toggled – for example when single-selection mode is active, all spatial objects that do not collide with picking ray are marked as not selected and removed from the list (this won’t be necessary in multi-selection mode) . Inside this function it is also determined what is considered a good pick result (ray colliding with an AABB? or OBB? and so on…) and this value is later used in traversal functions. Selection box parameters like position and dimensions also need refreshing – rectangle used for drawing uses bottom left corner as origin, just like the drawing methods provided with libGDX, however input event reporting uses the top left corner as the origin (this coordinates are fixed in picker position update function).

traverse() function definition

This function (when called properly) checks all the objects for ray intersections and updates the list of selected objects – also this is where the on-selection listeners are being called. Custom traverse procedure can be used, however one would still need to use begin() and end() functions (they’re required for the pick selection to work properly; begin() is the function where internal flags are set and checked).


Using selection box with mouse picking

Selection box can be used for selecting multiple objects at once. It is drawn by holding the mouse button (or selection button) and dragging the selector pointer. Depending on what the final usage is, the selection box can be drawn as a 2D rectangle on the screen or as a 3D rectangle on the scene plane or complex terrain. This demo uses the 2D solution for the selection box. When using the selection box which is covering some area of the screen, there is no way to use picking rays to determine which objects are selected (well one could cast rays from every covered pixel, but that approach would be far from optimal). The solution is to project corner points of the transformed AABB (so in fact an oriented bounding box) onto the screen and calculate the optimal 2D rectangle box for each spatial object. Then it will be possible to determine whether or not the selection box does overlap or contain any of the spatial objects on-screen rectangles. Please mind that this approach won’t be 100% accurate as sometimes the rectangles may be quite big and take more place than the 3D object on the screen. The animation below shows the accuracy of this method.

Using selection box with mouse picking
Using selection box with mouse picking

Setting up the framebuffer object

The framebuffer in OpenGL is a collection of multiple buffers that can be used as target for rendering without disturbing the main screen. A framebuffer object needs to be created for the user-defined framebuffer to work. Framebuffer object can also contain various images attached to it, depending on the usage. Images bound with color-renderable formats need to be attached to COLOR_ATTACHEMENT. This is where the fragment shader will output colors. Texture object will contain this image (or multiple images) and will be easily accessible from shaders via various methods. This is how rendering picture in picture effects work and other complex post-processing effects.

First the framebuffer object needs to be created by calling glGenFramebuffer() function and bound to the current context by using glBindFramebuffer() function – the first parameter determines the target to which the framebuffer will be bound.

Additional renderbuffer object needs to be created to store the depth information while rendering (this will allow to sort objects). Please note that textures or renderbuffer depth storage do not need to have the same size (resolution) as the screen/window. The framebuffer used for pick selection can be smaller to save memory and optimize speed – this will however cause the pick selection to be less accurate.

The internal texture to hold the image needs to have internal format of RGBA/RGBA8888. When rendering objects off-screen the custom shader will write single color values for each fragment – which are in fact 32 bit integers converted to four RGBA float components clamped to range 0.0 – 1.0.

As mentioned before, the color attachment texture and renderbuffer’s depth storage don’t need to have the same size as the screen. In this demo the default width is set to 256 pixels and height is proportional to the screen size.


Writing to and reading from the framebuffer

Custom rendering code needs to be used to update the selected framebuffer internal data. Also the second rendering pass needs to be done, now using different shader that just writes through the selected color passed via uniform variable. No shading nor lighting techniques are required, the rendered objects need to be absolutely flat having just one color. This color can be later read and converted to single integer value. Fortunately, the libGDX framework and Color class provides static methods for such conversions which makes things quite easier.

The additional class for rendering PickSelectionRenderer implemented for purposes of this demo won’t work with skinned meshes; The custom shader is also created using the DefaultShader object which contains one big shader with a lot of preprocessor conditions. When creating shader programs for use with the pick selection, the vertex shaders need to stay the same for given objects – only fragment shaders source code needs to be swapped.

The rendering code is based on the ModelBatch  class functions – the code uses Renderable objects which contain all information needed to make a single draw call – they define what (the shape), how (the material) and where (the transform) should be rendered by which shader program.

The PickSelectionRenderer class has access to the current PickSelection object instance along with PickSelectionFrameBuffer – this isn’t exactly a MVC model – it’s done this way so the CPU side pixel buffer is updated more easily and in optimal way. There’s no need to always copy all the pixels of the color attachment texture – just the size determined by the picking box (selection rectangle, if any). If it’s a single click selection, then only 1×1 pixel area needs to be copied to buffer and compared.

Also the PickSelectionRenderer implements the PickSelection.PixelChecker interface, so the PickSelection code can easily check the pixel data for given object index (object color).


Picking objects using the FBO texture data

In order to gain access to data stored inside of the color attachment texture, selected range of pixels need to be transferred from GPU memory. Easy optimization is to read only pixels covered by the selection box (other pixels won’t get tested anyway). Below is the source code for the function refreshing the pixel byte buffer (implemented in PickSelectionFrameBuffer class).

In the end of the render frame, the pick selection framebuffer needs to be updated. It only needs to be done when the picker is active (no need to run additional off-screen rendering if the pixel data won’t be used anyway). Additionally, object diffuse colors can be updated when necessary.

Pick selection color texture attachment contents
Pick selection color texture attachment contents


Complete source code of selected classes