Now, if you’re a hardcore 3D programmer, you’ll know exactly what to do with Molehill, and Thibault Imbert has a great introduction to the API for you. But if you’re anything like me, and your development experience has been in the world of 2D graphics or UI, you might find even this introductory material pretty head-scratching. Vertex and fragment shaders? Index buffers? Assembly language? LOLWUT?

I’ve just recently been learning more about GPU-based 3D programming myself, so I thought I’d try to make a molehill out of the 3D development mountain, and write an introduction to what this stuff is all about for those of us who are coming from the 2D world. In this first post, I’ll generally describe how modern GPUs work. I’m planning to write a follow-up post with more detail on how you actually work with the GPU for 3D graphics, and then another follow-up on how you can leverage the GPU for incredibly fast 2D graphics as well.

One caveat—I may say a few things that aren’t strictly true, mostly because I might be deliberately oversimplifying, but also because I might just be ignorant. Overall, I don’t think this picture of the world is too misleading, but please feel free to correct me in the comments.

Update: One fundamental point I meant to make when I originally wrote this post, but forgot to add, is that Molehill rendering is completely separate from display list rendering. All of the drawing that Molehill does basically ends up in a single layer that essentially draws into the background behind all of your display list content—the two don’t interact at all. I’ll discuss how you can leverage Molehill for 2D in a future post.

GPUs are scary and complicated!

That was certainly what I assumed before I started reading up on them. It’s true that developing high-quality performant 3D renderers using a GPU requires a lot of cleverness and math. But it turns out that you can describe what modern (programmable) GPUs do very simply:

1. GPUs draw triangles really, really fast.
2. GPUs do basic matrix and vector arithmetic really, really fast.
3. It’s up to you to tell the GPU exactly what to do with those capabilities.

That’s pretty much it. At the end of the day, modern GPUs are basically just really fast at drawing triangles and doing certain kinds of math. So fast that, unlike the Flash-style rendering you’re used to, you don’t program it like a display list, with minimal updates when things change. Since even a tiny change in your camera location or viewing angle means that all of the objects in your world are in slightly different places on the screen and with slightly different perspective relative to your view, you generally just tell the GPU to redraw the whole screen every frame. (Of course, there’s lots of optimizations you should do to make this work faster…but fundamentally you’re redrawing a lot more stuff than you would normally think of doing in the CPU-based 2D world.)

So what’s the big deal? I can draw a triangle really fast too.

The real power of the GPU is in how you tell it to draw those triangles and what math you tell it to do. Conceptually, the model (for programmable GPUs, which is what Molehill supports) is actually pretty simple, which is why the Molehill API only has a couple of dozen methods. It’s using the model effectively that gets complicated, as we’ll see later.

Here’s the basic way you work with a programmable GPU for a standard 3D rendering scenario. In each frame:

1. Triangles. Send the GPU a bunch of triangles, expressed in terms of their vertices, as well as other associated data like texture coordinates (which map vertices to locations in a texture bitmap).
2. Textures. Send the GPU a bunch of bitmaps to be used for textures that will be mapped onto triangles in step 4 below.
3. Vertex shader. Upload a “vertex shader” program, which does some math on each vertex from step 1 in order to produce a final vertex position, as well as other optional outputs that are up to you.
4. Fragment shader. Upload a “fragment (or pixel) shader” program. The GPU does built-in math to interpolate between the vertices on each triangle so it can call your pixel shader on each pixel in each triangle. Your shader does some math on each pixel, accessing textures from step 2 as necessary, in order to figure out what color that pixel should be.
5. Z-buffer. The GPU has a “Z-buffer” that stores the depth of each pixel it draws on the screen. If a pixel from a new triangle would be behind the last pixel that was written to the same screen location, it doesn’t redraw the pixel.
6. Lather, rinse, repeat. Repeat the above steps one or more times per frame.
7. Present. Once you’re done with all your drawing for the frame, you swap the buffer the GPU has been drawing to out to the screen, and it’s presented to the user. Now you’re ready to clear the buffer and start all over again for the next frame.

And that’s it. Simple, right?

Wait a minute. What about colors, curved surfaces, lights, reflections, cameras, perspective projection, and so on?

The GPU doesn’t know anything about any of that. This is the fundamental difference between older, “fixed-function” GPUs and modern programmable GPUs: the GPU makes very few assumptions about the actual meaning of all the data it’s processing. It knows about vertices, triangles, and pixels (or “fragments”, which are so-called because the edges of the triangle divide the pixels they cross into fragments). It also knows how to sample locations from texture bitmaps. Everything else—all the computation that actually decides which triangles to display, where the triangles should end up relative to the camera, or what colors the pixels should be given the current lights and textures—is either done by your application code on the CPU before you start uploading stuff to the GPU, or by the vertex and fragment shader programs that you upload to the GPU.

This means that you have complete power over how the GPU processes your geometry and textures. But with great power comes great responsibility—and a lot of coding, both in your application and in the shader programs you have to write to get anything to display.

Fortunately, you won’t have to write all that code yourself. Many members of the Flash community have written 3D frameworks and are now porting them to the Molehill API—including Alternativa, Away3D, Flare3D, Minko, Sophie3D, Yogurt3D, and more. In these frameworks, you don’t deal directly with the GPU, shaders, etc. at all. Instead, you typically interact with a “scene graph”, which is analogous to the Flash display list. It’s a persistent tree of objects that you can add and remove and set properties on. You just specify what geometry you want, how it should look, and where it should go in the world, and the framework takes care of sending it to the GPU and providing the appropriate shaders.

If the world of GPU programming doesn’t look like it’s for you, stop reading right now and go check out all those great frameworks. If you feel like digging in at a lower level, though, check out my next blog post, which will get into more details that bridge the gap between the high-level GPU description I gave above and Thibault’s API introduction.

Because this is my blog, I’ll flog my own session first: I’ll be running a “Bring Your Own Laptop” (BYOL) lab on Building Mobile Flex Applications. The way these labs work is that you, er, bring your own laptop, and we give you bits to install on it for the session. Also, if you have an Android 2.2 phone, bring it (and the phone’s USB cable) along too–though that’s not required for this lab.

Glenn Ruehle and Chiedo Acholonu from the Flex Mobile team will be presenting Deep Dive into Mobile Development with the Flex SDK, which will go into detail about the mobile development features we’re adding in “Hero”, and how to optimize your development process as well as your applications when building across multiple screens.

Ryan Frishberg, another engineer on Flex Mobile, will be presenting a session on Performance Tips and Tricks for Flex and Flash Development. This session will discuss optimizing both desktop and mobile applications, and describe common performance problems that developers of every level can encounter when building Flex and Flash applications.

Evangelist Christophe Coenraets will be presenting both a session and a BYOL lab on building data-driven mobile applications with Flex “Hero” and AIR for Android. His sessions will be focused more on the data connectivity side rather than the basics of mobile application and UI development.

And from outside the building, we have RJ Owen and Juan Sanchez from EffectiveUI presenting Screens of Possibility: Pushing Multiscreen Experiences with Spark and Flex, discussing their experiences designing and developing on desktop and mobile devices. I can’t reveal details of the app they’ll be showing off in this session, but I’ve seen some early comps and I think it’s going to be very cool.

Of course, there’s going to be plenty more at MAX about mobile development, Flex, and AIR; check out the MAX Session Catalog for the complete list. Hope to see you there!

If you’ve been following Flex over the last year or so, you might have heard of a related effort, code-named “Slider”, which was going to be a separate version of the framework that would be specifically optimized for mobile devices. What we’ve found over the past year is that vast improvements in Flash and AIR runtime performance on mobile devices, as well as the rapidly increasing power of modern touchscreen smartphones, have made it unnecessary for us to split off a separate framework. So going forward, we’ll have a unified framework for both desktop and mobile development, and “Hero” will be the first fruit of this approach.

For a good overview of our plans for mobile development, take a look at the Flex and Mobile whitepaper, FAQ, and Adobe Labs page. We’ve also started to post draft specs for the mobile features (as well as the rest of Hero) to the Flex Open Source site, and we’ll be rolling out more over time.

I’m excited that we’ve started talking about our mobile Flex efforts publicly, and look forward to revealing more over the next few months. Watch this space!

Various people have requested the source code, so I’ve decided to host it on Google Code at http://snackr.googlecode.com/. You can file bugs/enhancement requests on the Issues tab there (the issues and wish list items from snackr.net have been migrated there), and check out the source anonymously using Subversion (instructions are on the Source tab).

The source is currently in an interim state–there are a couple of half-implemented features (the ability to “star” items to read later, and the Google Reader integration that Rob is working on)–but you should be able to run what’s there (the GR stuff isn’t hooked up at all to the UI yet, and the “star” functionality works but the UI is slightly broken). I’m planning to post interim test builds there once in awhile as well.

The source also probably isn’t the prettiest code ever , and I haven’t finished commenting it, but I’m hoping to continue cleaning it up and refactoring over time. Feel free to check it out, and let me know if you have suggestions–or, better yet, patches you’d like to submit!

• Allow multiple selection in the feed list, so you can delete a bunch of feeds at once. It seems like a lot of people imported large feed lists from their existing blog readers, then realized they didn’t want to actually see all of those feeds in Snackr.
• Support Asian language characters. Currently, Snackr uses Myriad Web, and the font is embedded in order to make fade animations work properly for text, but that font only has Latin characters. I’ll either need to add an option to use the system font (and turn off fade animations), or maybe build a version that embeds a font with Asian characters in it (though that would probably lead to a huge installer).
• Ability to keep a list of items to read later (e.g. by starring them). I’ve been wanting this for awhile but haven’t gotten around to implementing it yet–now I have some incentive!
• Posting the source. I do really want to do this, but I need to set aside some time to make the code slightly less embarrassing

I also noticed today as I was using it on my machine that for some short posts, the popup seems to “bobble around” a bit and end up at a very thin size (it’s different from the jittery animation on Vista/Linux; this is happening even on OS X). It’s intermittent, but it reproduces pretty regularly on certain kinds of posts. Has anybody else seen this? I’ll have to look into it–it must be a recent injection, as I never noticed it before.

I’m planning to make regular updates to Snackr, though things are pretty busy; I’m hoping I can carve out enough time to release a new version in a couple of weeks. Snackr should automatically notify you when an update is available.

Keep those cards and letters coming!

Around the same time I came to this realization, Adobe AIR 1.0 was publicly released. I wanted to try to write an AIR app just for fun, and it occurred to me that I might be able to make something that would solve my RSS problem.

The result is Snackr, a ticker-like widget that lives on the bottom (or side) of your screen and scrolls random items from your RSS feeds. (It’s called “Snackr” because it lets you nibble on your feeds. Guffaw.) Here’s what it looks like on my desktop:

I’m actually finding Snackr really useful—it helps me keep up with blogs I want to keep up with, and also gives me a great smattering of items from sources I wouldn’t normally read regularly. Please try it out and let me know if you like it! (Of course, it’s still an alpha, so please expect bugs; there’s a list of known issues on the Snackr website.)

Snackr has also been really fun to write, and along the way I figured out some tips and tricks for doing various things with Flex and AIR. Some notes on that after the jump.

Snackr is written in Flex 3 (and, of course, AIR). I hadn’t done any AIR development when I started writing it, and it’s a testament to how well AIR is designed that I got the basic thing up and running in a weekend. Of course, most of that was due to my familiarity with Flex, but the fact that AIR makes it trivially easy to package up a normal Flex app as a desktop app is easy to take for granted. With almost no work, I went from being a Flex developer to a cross-platform desktop app developer.

Once I got the basic ticker UI up and running, I started exercising more AIR-specific functionality. Initially, I kept all the feed items in memory, but decided to try out AIR’s local database capability to improve Snackr’s memory footprint. Like a lot of other things in AIR, it was dead easy to set this up; I’m not a database/SQL guru, but it was very straightforward to build out the database code. (I used synchronous mode for simplicity; it would have been much more complex had I tried to do it in asynchronous mode—as with all other Flash/Flex/AJAX programming, asynchronicity is one of the toughest things to manage cleanly, especially if you have a chain of things you want to happen in sequence.)

The next thing I did was build the animation for the popup windows. I ran into a couple of little snags here. I started by trying to just animate the Flex Window component using standard Flex effects, but that didn’t seem to work, probably because setting (say) the x and y of the window component doesn’t actually set the position of the underlying AIR NativeWindow. So, to get it to work correctly, I directly animated the “bounds” property of the underlying AIR NativeWindow class. In order to keep both the position and size of the bounds in sync as I animated, I wrote my own AnimateBounds effect—this was easy to achieve by subclassing TweenEffect and just converting the bounds property of NativeWindow to an array of numbers as TweenEffect expects.

I also had to be careful when drawing the “bubble” background in my popup window’s updateDisplayList() function. Before I added the animation, the background drew fine, but I found that during the animation it didn’t seem to redraw at the right size at each step. I discovered that I had to ignore the standard unscaledWidth/unscaledHeight being passed into updateDisplayList(), and (once again) just look directly at the AIR NativeWindow bounds to get the width and height to use. I’m guessing that during fast animations, the bounds of the Flex Window component don’t update synchronously with the NativeWindow bounds.

Around the time I was finishing up the animation stuff, we released the alpha version of AIR for Linux. Just for grins, I installed Ubuntu in VMWare on my Mac and tried it out—and amazingly enough, it worked! Well, with one snag. As Linux folks know, window transparency is something that’s only recently been supported in Linux, and not all window managers have it. I couldn’t figure out how to get a transparency-supporting window manager set up, so I just punted; the popups look okay without it (though they look like rectangles rather than bubbles).

There was a bigger issue, though. I noticed that unlike on Mac OS X and Windows XP, the popup animation was really jittery on Linux. Later on, when I tried Snackr out on Windows Vista, I noticed some jitter there too (although it’s not nearly as bad as on Linux). I haven’t yet figured out exactly what causes this—I filed a bug with the AIR team—but I’m guessing that on Vista and Linux, even when I set both the position and size of the popup at the same time, the OS either (1) processes them separately, so they get out of sync, or (2) processes the move immediately by directly blitting the window at its current size before I get a chance to redraw it on the resize event. (This mostly comes into play because I’m actually animating the popups upwards, so both the position and size have to change at the same time. It wouldn’t be an issue if you’re just growing a window larger or smaller but keeping it at the same position.)

The final issue, and probably the biggest one, is that there’s no way currently to have Snackr act like a true docking window—it can’t tell the OS to reduce the available space of the desktop, so if you maximize a window, the window will overlap Snackr. There isn’t currently an API for this in AIR, so there’s no real way to work around this for now. To compensate, I’ve added a bunch of convenient ways to get Snackr out of your way; you can toss it around the screen to different sides, put it on a second monitor, or collapse it temporarily.

Other than that, all the functionality I’ve put in Snackr has been really easy to implement. AIR makes it really simple to support multiple monitors, for example. And it’s also pretty easy to implement auto-update—you have to set up whatever logic you want to figure out when a new version is available (e.g. by downloading an XML file from a website), but once you retrieve a newer installer it’s a single function call to run the updater to install it.

I’m planning to put up the source somewhere eventually, but I want to clean it up a bit first (yes, my coding habits are sloppier at home than at work). I definitely plan to continue to add features to it—since I’m using it myself, I have a lot of incentive to keep it up. Also, my colleague Rob Adams (who was the first person to use Snackr besides myself, and has given me lots of great feedback) is planning to add some features to it as well—notably, making it synchronize the feed list and read items with Google Reader, so you can use it as an adjunct to your normal blog reading habits instead of a replacement. (Please feel free to bug him if you want this feature.)

Enjoy Snackr—I’d love to get your feedback!

]]> http://www.rictus.com/muchado/2008/05/12/snackr-an-rss-ticker-built-using-air-and-flex/feed/ 23 http://www.rictus.com/muchado/2008/04/23/video-interview-with-me-about-thermo/ http://www.rictus.com/muchado/2008/04/23/video-interview-with-me-about-thermo/#comments Wed, 23 Apr 2008 22:01:07 +0000 nj http://www.rictus.com/muchado/?p=38 It’s been busy here in Thermo-land, as we work furiously (no really! we’re furious people!) to turn our vision into reality. I have a fun side project that I’m going to post about in a little bit once I get a few bugs worked out. In the meantime, here’s a video interview that Ryan Stewart, Thermo evangelist extraordinaire, did with me about designer/developer workflow in Thermo and Flex 4. It was our first video, so it’s a little blurry and off-center, but just pretend it’s artsy and edgy and you’ll be fine. (We did edit out the part where the siren went off when someone went out the wrong door in the cafeteria.)

Since we did the demo, I’ve been semi-obsessively searching MXNA for blog posts about Thermo, and among the generally positive responses, people have also posted a number of questions and concerns. I thought I’d address some of these here, to help clarify and amplify what we showed in the demo. (Sorry–no spoilers about release dates or other features here!)

The demo showed Thermo creating a lot of bitmap graphics. Will Thermo applications be large and bitmap-heavy?

Why is it useful to have graphics tags? Why not just import graphics as SWFs or bitmaps as in earlier versions of Flex?

The text field that was created in the demo seemed to have its skin specified inline. Will Thermo create large single-file applications? Will the code be huge?

It’s great that you can turn graphics into buttons, scrollbars, etc., but what about custom components? Will Thermo work with components created by developers?

Thermo seems geared towards letting designers build a complete Flex application. Does that mean you expect designers to create business logic as well?

That’s all for now. Please feel free to comment (or post in your own blogs) if you have more questions about Thermo–naturally, we can’t answer all of them at this point, but we’re really interested to hear your thoughts!

In the meantime, one bit of news–the Code Anatomy prototype I published in my last post is turning into a real feature! Our Learning Resources team is getting a real Flex developer to rebuild it from the ground up as a skinned, fully-reusable Flex app, and will be shipping several code samples that use it with Flex 3. Best of all, we’re planning to release the source code, so other folks can post anatomies of their own apps.

I probably won’t be coming up for air till after MAX (at least), but I will be attending, though not giving a talk. Hope to see you there!