• multiscreen-dev-with-flex-360flex-2011.pptx, if you have PowerPoint
• multiscreen-dev-with-flex-360flex-2011.pptx.pdf, if you don’t

I actually recommend looking at the PDF (which has annotations for most non-textual slides describing what I talked about on each slide) first, because the PPT uses an Adobe corporate font that you might not have, so the text might not look right. However, a couple of the diagrammatic slides make more sense if you actually go through the animation builds in the PPT.

Also, since I suck at this whole “social media” thing, I forgot to mention that I’m @rictus on Twitter–feel free to follow me there; I don’t post a lot, but do pop up there occasionally.

I’m still working on cleaning up the sample app from the top and fixing a few bugs–once it’s ready (hopefully within the next week) I’ll post a link to the source here as well.

Now, if you’re a hardcore 3D programmer, you’ll know exactly what to do with Molehill, and Thibault Imbert has a great introduction to the API for you. But if you’re anything like me, and your development experience has been in the world of 2D graphics or UI, you might find even this introductory material pretty head-scratching. Vertex and fragment shaders? Index buffers? Assembly language? LOLWUT?

I’ve just recently been learning more about GPU-based 3D programming myself, so I thought I’d try to make a molehill out of the 3D development mountain, and write an introduction to what this stuff is all about for those of us who are coming from the 2D world. In this first post, I’ll generally describe how modern GPUs work. I’m planning to write a follow-up post with more detail on how you actually work with the GPU for 3D graphics, and then another follow-up on how you can leverage the GPU for incredibly fast 2D graphics as well.

One caveat—I may say a few things that aren’t strictly true, mostly because I might be deliberately oversimplifying, but also because I might just be ignorant. Overall, I don’t think this picture of the world is too misleading, but please feel free to correct me in the comments.

Update: One fundamental point I meant to make when I originally wrote this post, but forgot to add, is that Molehill rendering is completely separate from display list rendering. All of the drawing that Molehill does basically ends up in a single layer that essentially draws into the background behind all of your display list content—the two don’t interact at all. I’ll discuss how you can leverage Molehill for 2D in a future post.

GPUs are scary and complicated!

That was certainly what I assumed before I started reading up on them. It’s true that developing high-quality performant 3D renderers using a GPU requires a lot of cleverness and math. But it turns out that you can describe what modern (programmable) GPUs do very simply:

1. GPUs draw triangles really, really fast.
2. GPUs do basic matrix and vector arithmetic really, really fast.
3. It’s up to you to tell the GPU exactly what to do with those capabilities.

That’s pretty much it. At the end of the day, modern GPUs are basically just really fast at drawing triangles and doing certain kinds of math. So fast that, unlike the Flash-style rendering you’re used to, you don’t program it like a display list, with minimal updates when things change. Since even a tiny change in your camera location or viewing angle means that all of the objects in your world are in slightly different places on the screen and with slightly different perspective relative to your view, you generally just tell the GPU to redraw the whole screen every frame. (Of course, there’s lots of optimizations you should do to make this work faster…but fundamentally you’re redrawing a lot more stuff than you would normally think of doing in the CPU-based 2D world.)

So what’s the big deal? I can draw a triangle really fast too.

The real power of the GPU is in how you tell it to draw those triangles and what math you tell it to do. Conceptually, the model (for programmable GPUs, which is what Molehill supports) is actually pretty simple, which is why the Molehill API only has a couple of dozen methods. It’s using the model effectively that gets complicated, as we’ll see later.

Here’s the basic way you work with a programmable GPU for a standard 3D rendering scenario. In each frame:

1. Triangles. Send the GPU a bunch of triangles, expressed in terms of their vertices, as well as other associated data like texture coordinates (which map vertices to locations in a texture bitmap).
2. Textures. Send the GPU a bunch of bitmaps to be used for textures that will be mapped onto triangles in step 4 below.
3. Vertex shader. Upload a “vertex shader” program, which does some math on each vertex from step 1 in order to produce a final vertex position, as well as other optional outputs that are up to you.
4. Fragment shader. Upload a “fragment (or pixel) shader” program. The GPU does built-in math to interpolate between the vertices on each triangle so it can call your pixel shader on each pixel in each triangle. Your shader does some math on each pixel, accessing textures from step 2 as necessary, in order to figure out what color that pixel should be.
5. Z-buffer. The GPU has a “Z-buffer” that stores the depth of each pixel it draws on the screen. If a pixel from a new triangle would be behind the last pixel that was written to the same screen location, it doesn’t redraw the pixel.
6. Lather, rinse, repeat. Repeat the above steps one or more times per frame.
7. Present. Once you’re done with all your drawing for the frame, you swap the buffer the GPU has been drawing to out to the screen, and it’s presented to the user. Now you’re ready to clear the buffer and start all over again for the next frame.

And that’s it. Simple, right?

Wait a minute. What about colors, curved surfaces, lights, reflections, cameras, perspective projection, and so on?

The GPU doesn’t know anything about any of that. This is the fundamental difference between older, “fixed-function” GPUs and modern programmable GPUs: the GPU makes very few assumptions about the actual meaning of all the data it’s processing. It knows about vertices, triangles, and pixels (or “fragments”, which are so-called because the edges of the triangle divide the pixels they cross into fragments). It also knows how to sample locations from texture bitmaps. Everything else—all the computation that actually decides which triangles to display, where the triangles should end up relative to the camera, or what colors the pixels should be given the current lights and textures—is either done by your application code on the CPU before you start uploading stuff to the GPU, or by the vertex and fragment shader programs that you upload to the GPU.

This means that you have complete power over how the GPU processes your geometry and textures. But with great power comes great responsibility—and a lot of coding, both in your application and in the shader programs you have to write to get anything to display.

Fortunately, you won’t have to write all that code yourself. Many members of the Flash community have written 3D frameworks and are now porting them to the Molehill API—including Alternativa, Away3D, Flare3D, Minko, Sophie3D, Yogurt3D, and more. In these frameworks, you don’t deal directly with the GPU, shaders, etc. at all. Instead, you typically interact with a “scene graph”, which is analogous to the Flash display list. It’s a persistent tree of objects that you can add and remove and set properties on. You just specify what geometry you want, how it should look, and where it should go in the world, and the framework takes care of sending it to the GPU and providing the appropriate shaders.

If the world of GPU programming doesn’t look like it’s for you, stop reading right now and go check out all those great frameworks. If you feel like digging in at a lower level, though, check out my next blog post, which will get into more details that bridge the gap between the high-level GPU description I gave above and Thibault’s API introduction.

Download the workbook (PDF)
Download the tutorial assets (ZIP)
Download the tutorial slides (PDF)

One of the great things about mobile development in Flex and Flash Builder is actually that it’s not all that different from developing desktop or web applications. When building mobile phone or tablet applications in Flex, you use the same Spark components you’re already familiar with; we’ve added mobile skins and touch interaction to the existing core components like Button and List. On top of that, you can take advantage of new components we’ve specifically created to make it easier to build mobile applications with the standard view-to-view navigation people are used to on touchscreen smartphones.

Similarly, building a mobile application with Flash Builder uses exactly the same workflow as building a desktop application. You create a mobile Flex project using the “New Flex Mobile Project” wizard, but once you’ve done that, you’re writing code and using design view the same way you would for any other Flex application. In a mobile project, you can choose to run/debug either on the desktop or on an actual attached physical device. If your device is on your WiFi network, you can even use all the standard Flash Builder debugging features on the device, like setting breakpoints, stepping through code, inspecting variables, and moving around the call stack.

When you’re done, you use Export Release Build to build an AIR application package that’s ready for deployment to devices. Unlike on the desktop, mobile AIR apps are packaged and installed the same way as native apps, so you can take the output of Export Release Build and send it directly to an app store, like the Android Market. To end users, AIR apps look just like any other application in the app store; the only difference is that the first time a user runs an AIR app, s/he is prompted to download the AIR runtime. After that, AIR applications install and run just like native apps.

We recently released AIR 2.5 for Android, and as we build and optimize AIR for other platforms, we’ll be bringing Flex and Flash Builder to those devices as well; BlackBerry Tablet OS and Apple iOS support are on the way, with even more platforms to come in the future.

Here’s a bunch of information to get you started building mobile Flex apps with the preview release:

• Read my ADC article, Mobile development using Adobe Flex SDK “Hero” and Flash Builder “Burrito”
• Check out my Flex Mobile videos on Adobe TV: part 1, part 2, part 3
• Download the Flash Builder “Burrito” preview release (includes Flex SDK “Hero”) from Adobe Labs
• Get started with tutorials and samples of mobile Flex applications

Of course, since this is still an early preview, there are bugs, performance issues, and missing features left to implement. But as you can see from the sample applications, you can do a lot with what’s there already. Let us know what you think–good or bad–in the Flash Builder forum.

I’m really excited that we’re finally getting to release this stuff to the general public. I know that the first time I got a Flex application running on an actual phone and hit a breakpoint on the device, it felt like a revelation. I hope it feels that way to you too!

Because this is my blog, I’ll flog my own session first: I’ll be running a “Bring Your Own Laptop” (BYOL) lab on Building Mobile Flex Applications. The way these labs work is that you, er, bring your own laptop, and we give you bits to install on it for the session. Also, if you have an Android 2.2 phone, bring it (and the phone’s USB cable) along too–though that’s not required for this lab.

Glenn Ruehle and Chiedo Acholonu from the Flex Mobile team will be presenting Deep Dive into Mobile Development with the Flex SDK, which will go into detail about the mobile development features we’re adding in “Hero”, and how to optimize your development process as well as your applications when building across multiple screens.

Ryan Frishberg, another engineer on Flex Mobile, will be presenting a session on Performance Tips and Tricks for Flex and Flash Development. This session will discuss optimizing both desktop and mobile applications, and describe common performance problems that developers of every level can encounter when building Flex and Flash applications.

Evangelist Christophe Coenraets will be presenting both a session and a BYOL lab on building data-driven mobile applications with Flex “Hero” and AIR for Android. His sessions will be focused more on the data connectivity side rather than the basics of mobile application and UI development.

And from outside the building, we have RJ Owen and Juan Sanchez from EffectiveUI presenting Screens of Possibility: Pushing Multiscreen Experiences with Spark and Flex, discussing their experiences designing and developing on desktop and mobile devices. I can’t reveal details of the app they’ll be showing off in this session, but I’ve seen some early comps and I think it’s going to be very cool.

Of course, there’s going to be plenty more at MAX about mobile development, Flex, and AIR; check out the MAX Session Catalog for the complete list. Hope to see you there!

If you’ve been following Flex over the last year or so, you might have heard of a related effort, code-named “Slider”, which was going to be a separate version of the framework that would be specifically optimized for mobile devices. What we’ve found over the past year is that vast improvements in Flash and AIR runtime performance on mobile devices, as well as the rapidly increasing power of modern touchscreen smartphones, have made it unnecessary for us to split off a separate framework. So going forward, we’ll have a unified framework for both desktop and mobile development, and “Hero” will be the first fruit of this approach.

For a good overview of our plans for mobile development, take a look at the Flex and Mobile whitepaper, FAQ, and Adobe Labs page. We’ve also started to post draft specs for the mobile features (as well as the rest of Hero) to the Flex Open Source site, and we’ll be rolling out more over time.

I’m excited that we’ve started talking about our mobile Flex efforts publicly, and look forward to revealing more over the next few months. Watch this space!

On Monday, we released betas of:

• Flash Catalyst — This is the new product I’ve been primarily working on for the last couple of years: the interaction design tool formerly codenamed “Thermo”. It allows you to take artwork created in the Creative Suite tools and turn it into a working Flash interface. You can publish it as a standalone piece, or give it to a developer to add functionality in…
• Flash Builder 4 — This is the next version of Flex Builder, with a slightly new name and a whole bunch of new features for developer productivity and easy data access. We renamed it Flash Builder in order to emphasize that it’s a tool for building all kinds of Flash applications (including pure-ActionScript apps), aligning it better with our other Flash Platform tools, and reserving the name Flex for…
• Flex 4 SDK — the open-source framework that’s the foundation of both Flash Catalyst and Flash Builder 4. The biggest feature is the new Spark component architecture, which completely separates the presentation of components from their underlying logic. Without Spark, Flash Catalyst wouldn’t be able to make it so easy to turn artwork into skinned components.

If you do try out Flash Catalyst, check out the great documentation, tutorials and videos page. In addition to content from our Learning Resources team, there are videos from some of the developers on the FC team, and from other folks on our prerelease group. It’ll really help you get up to speed quickly.

As someone who helped start this and this, I can say it’s always a little nerve-wracking to release the first public beta version of a new 1.0 product. You wince at every bug you knew someone was going to run into (and the ones you didn’t know about), and you wish you could just tell everyone “we’re planning to get to it, really!” for all the features that didn’t make the beta, or that you know aren’t even going to make the 1.0 release. But that’s dwarfed by the excitement of having people encounter it for the first time and seeing what they can actually make with it despite its limitations. I’ve already been amazed by what we’ve seen our prerelease group do with Catalyst, and I’m looking forward to see what you can do with it!

So, please back up your database and feed list, download v0.39 TEST and check it out! As always, you can leave comments here on issues that you find, file them at the issue tracker on Google Code, or send me email.

Also, I’m thinking of instituting an “official” beta program for Snackr. The requirements for participation are low:

• be willing to get announcement emails every so often
• download new test builds as they come out and use them for a few days
• answer a quick survey on each build saying how well the build is working for you

If you’re interested, please fill out the beta signup form. Thanks!