Now, if you’re a hardcore 3D programmer, you’ll know exactly what to do with Molehill, and Thibault Imbert has a great introduction to the API for you. But if you’re anything like me, and your development experience has been in the world of 2D graphics or UI, you might find even this introductory material pretty head-scratching. Vertex and fragment shaders? Index buffers? Assembly language? LOLWUT?

I’ve just recently been learning more about GPU-based 3D programming myself, so I thought I’d try to make a molehill out of the 3D development mountain, and write an introduction to what this stuff is all about for those of us who are coming from the 2D world. In this first post, I’ll generally describe how modern GPUs work. I’m planning to write a follow-up post with more detail on how you actually work with the GPU for 3D graphics, and then another follow-up on how you can leverage the GPU for incredibly fast 2D graphics as well.

One caveat—I may say a few things that aren’t strictly true, mostly because I might be deliberately oversimplifying, but also because I might just be ignorant. Overall, I don’t think this picture of the world is too misleading, but please feel free to correct me in the comments.

Update: One fundamental point I meant to make when I originally wrote this post, but forgot to add, is that Molehill rendering is completely separate from display list rendering. All of the drawing that Molehill does basically ends up in a single layer that essentially draws into the background behind all of your display list content—the two don’t interact at all. I’ll discuss how you can leverage Molehill for 2D in a future post.

GPUs are scary and complicated!

That was certainly what I assumed before I started reading up on them. It’s true that developing high-quality performant 3D renderers using a GPU requires a lot of cleverness and math. But it turns out that you can describe what modern (programmable) GPUs do very simply:

1. GPUs draw triangles really, really fast.
2. GPUs do basic matrix and vector arithmetic really, really fast.
3. It’s up to you to tell the GPU exactly what to do with those capabilities.

That’s pretty much it. At the end of the day, modern GPUs are basically just really fast at drawing triangles and doing certain kinds of math. So fast that, unlike the Flash-style rendering you’re used to, you don’t program it like a display list, with minimal updates when things change. Since even a tiny change in your camera location or viewing angle means that all of the objects in your world are in slightly different places on the screen and with slightly different perspective relative to your view, you generally just tell the GPU to redraw the whole screen every frame. (Of course, there’s lots of optimizations you should do to make this work faster…but fundamentally you’re redrawing a lot more stuff than you would normally think of doing in the CPU-based 2D world.)

So what’s the big deal? I can draw a triangle really fast too.

The real power of the GPU is in how you tell it to draw those triangles and what math you tell it to do. Conceptually, the model (for programmable GPUs, which is what Molehill supports) is actually pretty simple, which is why the Molehill API only has a couple of dozen methods. It’s using the model effectively that gets complicated, as we’ll see later.

Here’s the basic way you work with a programmable GPU for a standard 3D rendering scenario. In each frame:

1. Triangles. Send the GPU a bunch of triangles, expressed in terms of their vertices, as well as other associated data like texture coordinates (which map vertices to locations in a texture bitmap).
2. Textures. Send the GPU a bunch of bitmaps to be used for textures that will be mapped onto triangles in step 4 below.
3. Vertex shader. Upload a “vertex shader” program, which does some math on each vertex from step 1 in order to produce a final vertex position, as well as other optional outputs that are up to you.
4. Fragment shader. Upload a “fragment (or pixel) shader” program. The GPU does built-in math to interpolate between the vertices on each triangle so it can call your pixel shader on each pixel in each triangle. Your shader does some math on each pixel, accessing textures from step 2 as necessary, in order to figure out what color that pixel should be.
5. Z-buffer. The GPU has a “Z-buffer” that stores the depth of each pixel it draws on the screen. If a pixel from a new triangle would be behind the last pixel that was written to the same screen location, it doesn’t redraw the pixel.
6. Lather, rinse, repeat. Repeat the above steps one or more times per frame.
7. Present. Once you’re done with all your drawing for the frame, you swap the buffer the GPU has been drawing to out to the screen, and it’s presented to the user. Now you’re ready to clear the buffer and start all over again for the next frame.

And that’s it. Simple, right?

Wait a minute. What about colors, curved surfaces, lights, reflections, cameras, perspective projection, and so on?

The GPU doesn’t know anything about any of that. This is the fundamental difference between older, “fixed-function” GPUs and modern programmable GPUs: the GPU makes very few assumptions about the actual meaning of all the data it’s processing. It knows about vertices, triangles, and pixels (or “fragments”, which are so-called because the edges of the triangle divide the pixels they cross into fragments). It also knows how to sample locations from texture bitmaps. Everything else—all the computation that actually decides which triangles to display, where the triangles should end up relative to the camera, or what colors the pixels should be given the current lights and textures—is either done by your application code on the CPU before you start uploading stuff to the GPU, or by the vertex and fragment shader programs that you upload to the GPU.

This means that you have complete power over how the GPU processes your geometry and textures. But with great power comes great responsibility—and a lot of coding, both in your application and in the shader programs you have to write to get anything to display.

Fortunately, you won’t have to write all that code yourself. Many members of the Flash community have written 3D frameworks and are now porting them to the Molehill API—including Alternativa, Away3D, Flare3D, Minko, Sophie3D, Yogurt3D, and more. In these frameworks, you don’t deal directly with the GPU, shaders, etc. at all. Instead, you typically interact with a “scene graph”, which is analogous to the Flash display list. It’s a persistent tree of objects that you can add and remove and set properties on. You just specify what geometry you want, how it should look, and where it should go in the world, and the framework takes care of sending it to the GPU and providing the appropriate shaders.

If the world of GPU programming doesn’t look like it’s for you, stop reading right now and go check out all those great frameworks. If you feel like digging in at a lower level, though, check out my next blog post, which will get into more details that bridge the gap between the high-level GPU description I gave above and Thibault’s API introduction.

Okay, enough highfalutin’ chatter. This entry actually has (arguably) useful code in it.

The reflection effect is destined to become one of those gratuitous UI clichés, like brushed metal and gleaming highlights. But hey, the kids seem to like it, so I figured I’d try to make a similar component in Flex.

Before jumping into the explanation, here’s the demo:

• View the demo
• View the source
• Download the source (ZIP, 12K)

You can drag the panel around to watch the reflection follow it, and play with the sliders to tweak the look of the reflection.

I’m not the first to attempt this: Trey Long created a reflection effect in an early beta of Flex 2 (the demo pointed to from that link no longer works with the shipping version of Flash Player 9). However, his version created static reflections—so, for example, you could see the reflection of a Button, but if you moused over or clicked on it, the reflection wouldn’t update to reflect (no pun intended) the visual changes to the Button.

I decided to make the reflection “live”—as you interact with the components being reflected, the reflection actually updates in (near) real-time. I also componentized the reflector itself, so you can just target it at a component and it automatically positions itself appropriately. I used code very similar to Trey’s reflection filter to render the reflection bitmap, though the way I actually draw it to the screen is a little different.

Read on for an explanation of the code:

The reflection component is called Reflector, in the com.rictus.reflector package. If you take a look at the demo application, LiveReflection.mxml,
you’ll see that the invocation of the Reflector component is quite simple:

<reflector:Reflector target="{panel}" alpha="{alphaSlider.value}" falloff="{falloffSlider.value}"/>

Reflector is a UIComponent, so you can stick it anywhere a UIComponent goes. It defines two properties:

• target: the component you want to reflect. The reflector automatically tries to position itself below this component, so the target component should live inside an absolutely-positioned container (like a Canvas or a Panel with layout="absolute"), and the reflector should be inside the same container.
• falloff: a value between 0 and 1 that represents how much of the target component to reflect. 0 means to reflect none of the component, 1 means to reflect all of it, and values in between essentially set the ratio of the height of the component that will be reflected. In the demo, this defaults to 0.6.

In addition, you’ll also want to set the alpha value to a low number (it defaults to 0.3 in the demo). Play around with the sliders in the demo to get a feel for how these values affect the look of the reflection.

To see how it works, look at the source code for Reflector. Reflector listens for updateComplete events on its target, which fire every time the target’s updateDisplayList() method is called. The Reflector then updates its own display list to draw the reflection.

Here are some notes on the various methods involved.

set target(): This is where we register our listeners for events on the target. Note that when we register for the updateComplete event, we pass true for the useCapture argument to addEventListener(). This is necessary because updateComplete isn’t a “bubbling” event—it doesn’t automatically pass up to listeners that are set on ancestors of the updated item. For example, in the demo, if we didn’t get update events from descendants, you wouldn’t see the reflection update when you twiddle one of the sliders; you would only see it update if the outer Panel itself changed.

We can get around this by attaching a “capture” listener for the event; because of the way the Flash event mechanism works, this lets us see update events for all descendants of the target component, not just the component itself. I’ll have more to say about capture in a future entry.

handleTargetUpdate(), handleTargetMove(), handleTargetResize(): When the target is updated, we call invalidateDisplayList(), which tells Flex to redraw us (by calling updateDisplayList()) at the next available opportunity. If the target is moved or resized, we update our own position and size to match.

createBitmaps(): In order to avoid allocating and reallocating bitmaps every time we redraw, we allocate the bitmap storage once in createBitmaps(). We throw away these cached bitmaps whenever the target or the size of the target changes, but otherwise we can just keep redrawing into them. We also create the alpha gradient in createBitmaps(), since that gradient doesn’t need to change unless the target or its size changes:

updateDisplayList(): This is where we do the actual drawing of the reflection. The algorithm is pretty close to Trey’s original code:

1. Draw the target component into _targetBitmap.
2. Combine the target bitmap with the pre-calculated alpha gradient and blit the result into _resultBitmap.
3. Create a matrix transform to flip the image upside down.
4. Use beginBitmapFill() and drawRect() to blit the bitmap into the display list.

And that’s it! Now, some caveats.

• You may notice that I set showDataTips="false" on the sliders in the demo. I turned them off because the data tips don’t actually show up in the reflection (like vampires, I guess), since they’re not children of the Panel.
• I also set liveDragging="true" on the sliders. Without this, they don’t seem to send out updateComplete events as they’re being dragged. (One glitch in the demo is that if you click on the slider track rather than dragging it, the reflection doesn’t update as the slider animates to its new position, presumably because update events don’t get sent out during the animation even if liveDragging is set.)
• This isn’t visible on the black background, but normally Panels have a small dropshadow around them. That dropshadow doesn’t get drawn into the reflection; I’m guessing that BitmapData.draw() doesn’t render bitmap filters that are attached to the target component.

In a future entry, I’ll describe the draggable panel component that’s included in this demo.

Please let me know if you find a bug, or have any questions about how it works.

(Addendum: Someone noted that this code is GPL’ed. I did this because Trey’s original code was GPL’ed, and I didn’t go through the motions of reinventing the wheel. Future code I publish will likely be under some flavor of Creative Commons license, for easier reuse.)