Modern Desktop – prototyping Jerry’s Windows 9 vision


image

I recently came across a design from one Jerry Jappinen for modernizing the “desktop” ..

I thought it would be fun to take his designs to the next level and prototype it as a WinRT-XAML-C#-SharpDx APP.

Also the goal with this post is to share my thought process on how I would go about architecting such a design.

 

The Design

 

Jerry came up with some nice ideas for the “modern” windows desktop ..

image

 

image

image

 

And Jerry’s “hopes and dreams” for Windows 9 …

image

 

** I personally want to call out to Jerry to thank him for sharing. Its contributions like his that makes coding up today’s prototype worth while … :)

 

 

My Initial Breakdown of the Design

 

The main elements of the designs that I can make out are …

 

image

 

 

image

 

image 

 

The finished Prototype (still images)

 

Jumping ahead so that you can see what the end prototype looks like.

(there is a video of it running at the end of the post, these still images are not reflective of the prototype as all the animations are not visible)

 

image

 

image

 

image

 

Technologies I will use

 

 

 

Project layout

 

Generally for quick prototyping I like to have a simple folder structure for my assets/content/code .. The idea is to rapidly prototype ideas!

 

image

 

  • Assets – images, videos, general visual content
  • Controls – reusable UI components.
  • Converters – Convert data into something that the UI can use
  • Messages – MVVMLight Event Aggregator message definitions
  • Models – General data objects that is reusable and often basic types
  • Services – generic services reusable across the view models
  • ViewModel – the implementation logic behind views, generally has the “brains” of how a UI behaves and sometimes looks.
  • Views – contains the UI of the app, majority of UI is bound to VM’s and rarely do we use code behind

 

Background

 

I always tend to start with getting the background up and running for all my projects, and the main reason is i like to be visually immersed in my app from day one. I tend to do F5 (run my app) quite a lot during the course of the day so having a stimulating background makes it so much easier to keep my creative juices flowing :)

 

The goal with the background will eventually be to render it via Direct2D and incorporate rich effects and animations. For now it will only be an image and possibly rotating between a couple of them.

 

image

 

This background control renders in the MainPage.xaml and is z-ordered at the very bottom of the visual tree underneath all the other visual elements

 

image

 

 

User

 

The user details of the app consists of a View and View-Model . List most of the UI in the rest of the prototype i use MVVMLight to wire up the two.

 

image

 

image

And as you can see the important bits of the UI is filled in via binding, even thou its a prototype i find it faster to knock out the V’s and VM’s

image

 

And with most of my ViewModels i always have “sample” data that is visible at design time and runtime!

 

image

And as you can see BLEND has the sample data at design time that makes styling so much easier

image

 

The rest of the components share a similar architecture, V-VM , sample data, all important bits are bound! I won’t go into too much details for the future controls, i will however highlight interesting things!

 

Taskbar

 

The important things to know about taskbar are ..

 

1. Taskbar is split up into 3 different sections (tasks, status, time) and each is layed out and flows nicely so in the future when we have different device sizes and orientations these areas can nicely flow

image

 

2. The basic unit is a TILE … These Tiles are reused all over the place and this tile is as lean as possible

image

 

3. Where possible I try to use VECTOR assets rather than bitmaps. ALL the icons in my prototype are PATHs that i took from MetroStudio and bound via the VM’s to the Views!

 

image

One of the major reasons i like using vectors is when the user ZOOM’s into my UI they retain high fidelity …

 

4 .When tiles animate in I try to use “Transitions” over custom storyboards or custom code ..

image

 

 

 

Windows

 

A window consists of a Chrome and the Content. Ill cover the content below in the “App” section..

image

As each “Tile” in the taskbar is clicked a Window is created and placed in a “Windows” collection and rendered via binding to the View. Its important to know that its ALL about binding ViewModel’s to Views as much as humanly possible

image

 

 

 

Windowed Apps (eg. YouTube)

 

Now an APP that loads within a Window’s “CONTENT” area is very interesting … The approach I’ve taken is that the XAML for the APP is defined as PURE XAML with NO CODE BEHIND.

And more importantly this APP’s XAML is “dynamically” loaded in when needed.

In the case of the YouTube app, the only APP i’ve wired up to prove my point, is a pure XAML only UI…

image

This XAML is dynamically loaded into the VisualTree ONLY when needed, i.e. when the YouTube app tile is clicked.

image

 

The XAML has a binding to a ViewModel that currently is embedded within the prototype app, BUT ideally these VM’s will be separated into there own dlls and hopefully in the future possibly injected/MEFed in .. Cross fingers we are allowed to do this in the future!

image

 

The XAML View binds to the VM …

image

 

My goal is to have many different XAMLs that the USER can choose from for each APP. In the case of this demo YouTube app, a user in the future may choose from several different XAML skins , each of them bound to the same VM (YoutubeViewModel)..

 

Performance & Tuning

 

At every step of the way, as i created each control and F5d to see the running progress of my prototype, I made sure to continually check the taskmanager and do relevant memory watching.. I’ve learnt to proactively trace my app upfront rather than at the very end :)

I’m currently at 32mb footprint, quite happy with that ..

 

image

 

Other Info

 

  • All messaging between the View Model’s/Controls/Views/Services is done via Event Aggregator …
  • View-VM binding as much as possible , near zero code behind. Note i have no issue around codebehind, i just wanted to push as much logic into my VM’s so i can easily change views when i wanted to!
  • The SharpDx rich background will come in the next blog post
  • The SpinKit styles for loading animations are awesome… :)
  • Total time to prototype this was about 12hrs
  • The demo works targeting x86/64, ARM!

 

 

Video of the prototype

 

Here’s a quick 3min video of the running prototype on my Surface RT (1st generation). Because great care was taken to target ANY CPU i could easily target ARM :)

The second part of the video shows me running it natively on my laptop (Intel CPU).

 

 

 

Demo Code

 

You’ll need VS2013 to run the demo code ..

 

image

 

 

Conclusion

 

This first blog post was purely to get the architecture and the major XAML pieces done .. Next step is to do the SharpDx background ..

I have to admit that the tooling story has greatly improved since last i used it, windows 8.0 version of tooling. It was a pleasure to use Blend and VS2013, i had zero blend crashes :)

Even after all these years, WPF, Silverlight, Jupiter … i still honestly believe XAML is the best UI technology out there! :)

My modern “Start Screen” (Part 1)


image

I saw a design of a mocked up start screen on Deviant Art that I liked more than the one in Windows 8. It had gradients, shadows, 3D depth definitely went against some of the design goals of metro.

Below is what the current Windows 8 “start screen” looks like :

image

Below is the mocked up deviant art “start screen” design :

start_screen_unmetro_by_zainadeel-d4wszjj

What I liked about this is the use of gradients, shadows, depth to each tile ,… Pretty much all of those features are forbidden in current win8 metro design.

So I wanted to reproduce something similar using my favourite technologies XAML/C#/DirectX(SharpDx) ..

 

Design Analysis

 

image

Analysing the deviant art “tile” we see the following..

1. Depth to each tile, almost a 3D like effect, achieved using some form of “Drop Shadow” effect

2. Inner radial gradient for each tile that gives an inner glow like effect surrounding the icons in each tile

It’s a very artsy/pastel like overall effect .. I like it a lot Smile

So the question is how do we achieve this in WinRT XAML/C# ?

 

My proposed XAML/C# solution

When i see this design i immediately think Direct2D and Effect Graphs. The only way to get these shadow/Glow effects is using DirectX.

So my proposed solution would be to render it using SharpDx on a Direct2D surface utilizing effects. Further rendering this D2D surface in a XAML SurfaceImageSource (SIS) or SwapChainBackgroundPanel (SCBP).

image

Taking each of these parts lets create a tree of the UI proposed elements

image

Each tile will consist of the following visual elements that are rendered from bottom to top. This is the proposed “visual tree” of each tile.

 

Let’s Build it

 

As mentioned above I need to generate shadow/glow effects as well as a radial gradient so the only option for me is to use SharpDx.

There are numerous posts and examples of how to setup a SharpDx pipeline to render Dx content in XAML so I won’t be covering that here …

 

Step 1: Immediate Mode rendering requires me to maintain my own “visual tree”

In the XAML world, retained mode world, we are spoilt because a visual tree is maintained for us by the framework.

In the DirectX world, immediate mode world, we have to manage the rendering of each of the UI elements ourselves, including all the state between each frame!

I wrap all this logic in the Renderer classes…

image

 

I have a list that will hold the items for rendering “_renderTree”

image

 

Step 2: Preparing my assets

The assets for my tiles consists of

a) Background Images for the tiles

image

b) Icons for the tiles

image

 

All these assets will need to be loaded as textures and cached so that we don’t need to keep doing File IO reads.

To load a media element i have a method that wraps all that up in the BaseRenderer class. Its called “LoadAssetAsync”, and as the name suggests it loads it asynchronously. It uses the storage API’s to read the asset and then WIC to work with the bytes.

image

 

Once the asset is loaded from the file system i cache it for reuse later in case we need to re-render the entire _renderTree.

image

Loading assets and caching them for use as textures on a Direct surface is a very very difficult skill to learn, the only way to fully grasp this idea is to just throw yourself into the deep end and make many MANY mistakes Smile

Step 3: Drawing the Tiles

We have our assets, we have a way to create a render tree next step is to load that tree with our UI elements that use these cached assets!

I have a method that wraps up the creation of the Tiles

image

The actual logic to create a tile is wrapped up in another method called “_createTile”

This is where things get interesting as now i need to work with Effect Graphs

Step 4: Drawing a Tile

Fist step is to create a “BitmapSource” effect, this will be the bottom most image, the background image.

image

This BitmapSource effect will be used as input into an effect graph.

Note that i use a variable passed in to define the “Background Image” to use, if you step thru the code you’ll notice it goes through “step 2” above to cache the asset so if we reuse this effect, or re render this UI element it will use the cached version.

As you can see below if we render just the BitmapSource effect it is the FULL image, it does not look like a tile at all. And it doesn’t honour the width/height of the proposed tile dimensions (in this example a tile is 200px by 200px)

image

If you noticed from Step 2 above each of the backgrounds are oddly shaped and not the classic “tile” shape of a square, or rectangle (2 squares). We need to somehow scale the background to fit into the tile shape we want..

image

I used a direct2d “Scale” effect, and this scale effect acts on the “bitmap source” effect from above. Notice the * above. This is an effect graph Smile

Rendering this Scale effect, below is the result.

image

 

Now that the background image is scaled to fit the desired tile size we need to crop out the bits that fall outside. For this I use the “Crop” effect.

image

The input to the “Crop” effect is the previous “Scale” effect ( * above ). I should point out that the “Order” property above is used to order the layers for rendering, its like the z-index in web programming.

Rendering this “Crop” effect we get the 200px by 200px tile Smile

image

So we used an effect graph to draw the background of a tile.

 

Step 5: Rendering the inner glow

The inner glow is a rectangle with a radial gradient applied to it. Its pretty simple, and i’ve used this in previous demos Smile

image

It uses the RectangleGeometry in the SharpDx library

image

And the radial gradient applied

image

And rendered it looks like this:

image

 

 

Step 6: Rendering the outer shadow

The outer shadow is achieved using the “Shadow” effect. And the input to the shadow effect i’ll use the “Crop” effect from step 4 above. I could of used the scale or bitmapsource effect, as all i wanted was the outline of the tile.

image

The shadow effect rendered looks like this :

image

 

 

Step 7: icon & label

The Icon is rendered using a BitmapSource effect and to make it stand out I put a shadow effect around it . The reason i put a shadow around it is that if the background image blends too much into the icon it will be hard to see the icon.

image

This is the icon + shadow rendered:

image

The label is rendered using DirectWrite, ive wrapped the call into a method that does all the magic

image

image

 

Step 8: putting all the layers together

So rendering all the layers from step 4 to step 7 we get a tile Smile

image

Zoomed in :

image

Step 9: other useful tips

I used a “staging” texture to render the tiles so that when the user pans and zooms it uses this “staged” texture to render rather than re-rendering the entire renderTree.

image

When rendering the tiles i swap in my “staging” bitmap. Then the tiles are drawn in this staged target

image

After the tiles are finished drawing on the staged bitmap i swap back in the original render target

image

Now I have all my rendered tiles in a nice staged bitmap that i can manipulate when panning/zooming.

So the actual rendering to D2D uses this “Staged” bitmap ..

image

As you can see from above i also render “Debugging” info and a nice “Designer Surface Region” that represents the screens dimension.

 

 

Final Result (part 1)

Running the WinRT-XAML-C# demo after the initial load the frame rate settles to around 55-60 frames per second (fps) which is what I expected.

image

A zoomed in view of a tile we see the outer shadow, inner radial gradient etc. Exactly the effects i was going for Smile

image

 

Here’s a quick video of the demo running , its got a low fps due to screen recording and running simultaneously.

 

Sample Code

All my code can be found in my demo GIT project that i plan on using to also build my applications with.

image

Git Hub url : http://github.com/LiquidBoy/ModernApps

 

Conclusion

 

Part 1’s goal was to create tiles that have gradients, shadows and depth using an D2D effect graph. I pretty much achieved that Smile

The next step, part 2, will be to add the animations to the tiles and wire up some interaction between XAML and the D2D surface.

D2D Effect Graphs mixed with XAML is a very cool combination that i hope to master over time, i will admit thou it is a hard concept and skill to grasp. Patience and lots of reading is required, plus lots of reverse engineering samples!

I haven’t had this much fun in ages Smile , SharpDx gives us FULL access to DirectX / DirectWrite / WIC pretty much everything we need to create awesomeness from within XAML/C# ..

The fun continues with “part 2” where i try to animate all these tiles!

One possible future for Desktop?!


image

So last week, 25th march 2013 , a video leaked of someone using a version of Windows with a new UI/UX interaction for the start screen.

We believe it may be Windows Blue or possibly Windows 9?! or might be just MS internal developer’s/designers trying out some ideas ?!

Bottom line is it was an interesting new metaphor for desktop apps that was being shown off. Or more importantly a more connected experience between modern shell and desktop shell

UI/UX

The user starts in the start screen

image

They swipe bottom-to-top  to get to the apps list, incidentally in Windows Phone to do the equivalent is swipe right-to-left Smile [Windows learning more from Windows Phone, nice ! ]

image

image

Click on a desktop app (in this case a modernized SkyDrive desktop app)

image

 

At the moment the desktop skydrive app window is not in focus, click on the desktop to give the window/desktop focus (notice the window chrome has changed to that of being in focus)

image

At anytime swipe top/bottom edges to get to the AppBars (top/bottom), see how the top app bar can show a possible list of apps running on the desktop, notice the (+) as well.

The bottom app bar itself has interesting icons, one even resembles what you would see on the standard desktop task bar (the wifi network status icon). One can even assume that the classic desktop taskbar is nothing more than an app bar in this new modern world Smile

image

Notice in the classic Windows 7 taskbar they introduced those previews? Don’t they resemble what you would put in the top app bar ?! Smile

image

 

The bottom app bar , ill call it my task bar, in the leaked video can eventually get you back to the app list ..

image

This is part of the leaked video that shows the UI/UX experience

 

Actual Chinese link to video : http://www.win8china.com/html/4519.html

 

Conclusion …

This may be just internal designers/devs playing around with new Desktop / Modern UI metaphors ..

It could be MS’s answer to osx, linux etc multi-desktop views…

It’s interesting the idea of layering an modern app over the desktop itself that is window aware, that idea intrigues me.

This whole notion of mixxing the classic desktop as nothing more than a render layer within a modern app is something i want to explore ..

And this modern app layered on top of the desktop can give life to the desktop and the currently in focus window.. I like that idea!

I hope this video is for real, and the Modern Desktop UI/UX workflow is as well!

Smile

MVP Summit 2013


image

I’ve been an MVP for 5 years, the first year was as an Expression Blend MVP, the next 4 years were/are for Silverlight (I’m still an MVP)!

Going to the MVP summit is a massive trek for an Aussie like me, its a very painful 20+ hour flight(door-to-door) so for the last 4 years I passed on the trip.

However this 5th year I decided to go because like most of my Silverlight colleagues we felt it could be our last, Winking smile

Besides my fascination with Silverlight (all things XAML really) I’m also an avid follower of technologies that help me create crazy UI/UX experiences so delving into HTML/DirectX is not uncommon! And I also like to know the very low level details of the technologies I use, so hacking at the bits in the runtimes is part of what I do Smile

This is what I wanted to get out of the MVP summit…

 

1. Finding out about MS’s future strategies for XAML/HTML/DirectX, the UI stacks in Windows / Windows Phone / XBox and its other platforms.

2. Finding out about the .NET strategy .

3. Talk to experts in 1 & 2 above both from MS and from other MVP’s

I had a clear goal what I wanted to do on my trip!

 

And this is what I got out of the MVP Summit (and I’m being very careful not to break NDA here) …

 

1. put faces to twitter names, hung out and discussed cool stuff ! Invaluable!

2. “This info is off limits”, “Can’t talk about that” for a lot of the stuff I wanted to know about (XAML & WinRT) Sad smile

3. Confirmed my suspicions on the .NET strategy, confirmed what I was seeing in Windows Phone 7/7.5/8 and WinRT/.NETCore . There is a lot going on that has been brewing for many many years. We have been seeing it in Silverlight 3/4/5 and in Silverlight on Windows Phone. BUT it all made sense after some of the sessions and discussions with MS folk. Can’t really say much more !

4. Learned of 2 multi-year hush-hush projects that are as encompassing as Roslyn that explains a lot of what’s been going on with Silverlight

5. Saw that MS teams that adopted xaml (wpf / silverlight) early are in a great position to become agile going forward. These designers/developers are what i consider experts in XAML and can bend it to there wills, they are now in a great position to deliver agile UI/UX updates to us for there respective products!

6. SurfaceRT with TypeCover + OneNote is perfect for conferences, i must of typed the equivalent of 20 pages of notes..

7. The MS folks that were available to us were great people that wanted to be as open as possible BUT couldn’t. They really do care about the developer community BUT MS are at an inflection point with there tooling/frameworks/runtimes, and so they were not allowed to discuss such things!

8. This was the general tone of my introduction to people …

me: Hi

MVP 1: Hi, I’m david

me: Hey I’m Jose ..

MVP 1: ohhh (looking down at my tag)… your a Silverlight MVP …. <long uncomfortable pause> … im sorry!

me:  Smile its all cool!

 

9. Blend Tooling has been playing catch up for 3 years. Lots of refactoring and bringing in new platforms ,XAML-WinRT and HTML-CSS-WinJS, has caused Blend to stagnate in the innovative features category. They have smart people working on the blend tooling BUT seems like they still have about another year to go with all this convergence … Overall I liked what I “heard” from the Blend team and of the direction of it in VS .. Just sad that WPF/SL had to suffer! (note i said HEARD because we really didn’t see anything it was all discussions)

 

Summary

 

So in summary the top 5 topics / discussions / sessions that I found most interesting

5. Visual Studio vNext  …

4. TypeScript

3. Roslyn

2. JIT

1. BCL

 

Sorry I could not expand more on certain things (like 1 above Winking smile), NDA blah blah blah…

Know this, I am very excited about the direction of .NET.. Its had a great decade already and all signs point to an even better decade going forward!

Feels like MS are doubling down on .NET Smile

 

 

Jose Fajardo

Silverlight MVP

Lightning (SharpDx/XAML/C#/WinRT)


image

The other day I came across a very cool XNA demo, Michael Hoffman’s ‘How to Generate Shockingly Good 2D Lightning Effects’ .

I needed to redo this in my current favourite UI stack XAML/SharpDx/C# .

1 hour later i was done, thanks to the SharpDx toolkit it’s a breeze to bring across demos like this.

Here is the video of the Win8 lightning demo app :

 

Technologies used

1. SharpDx Toolkit – this is built ontop of SharpDx and it helps abstract out some of the hard concepts that can confuse new starters in the directx space. It shares a lot of ideas with XNA (Game/GraphicsDevice/Content Pipeline etc).

2. XAML – Using a Swap Chain Background Panel to render the Dx content

3. C#

4. WinRT – this is a modern app, uses the new API’s from the Windows team.

 

Interesting Techniques

1. Making SpriteFonts – I used MakeSpriteFont from the DirectX Toolkit to create the Lightning Font used in the Lightning Text.

2. SpriteFont to Point– the demo makes use of SpriteFonts, and getting the point information from a sprite and rendering each point as a lightning position. There are about 1000+ points created from the word “Lightning”, and its these points that are rendered in 3D using the Lightning technique.

3. 2 render targets to render the Lightning Text. The demo shows creating 2 render targets (currentFrame, lastFrame) that are used to push the 1000+ points and lightning textures on. We then push the currentFrame into the backbuffer of the device for rendering..

 

Sample Code

can be found on Skydrive

 

Final words …

I highly recommend looking at some of the techniques employed in this demo. Its a pretty cool lightning effect Michael came up with..

Have fun …

WinRT/XAML Colour Picker (using SharpDx)


image

I needed a colour picker for an app I’m building and it turns out no ones built a WinRT XAML one yet.

So this is my very simple colour picker!

I use bitmaps to represent the swatch (in my case i have 3 different types).

I use SharpDx to get the “Pixel Buffer” of the selected swatch

Where ever the mouse or pointer is i get the pixel under that position via the pixel buffer and from that i get the colours (RGBA)

This is what the colour picker looks like

image

 

This is a quick video of the colour picker :

 

Here’s the code

image

9 new XAML/SharpDx demos


image

[UPDATED : Demo code now works for FINAL RTM version of Windows 8 (9200) ]

I’m becoming more experienced with WinRT XAML and DirectX via SharpDx, and as such my demos are becoming more complex. This is all a learning process for me and the code reflects that. I hope you find something useful in the samples.

Here’s a quick video of the 9 new samples, I apologize upfront for the shaky camera work. If I try to screen capture on my tablet the CPU utilization negatively impacts the performance of the running app.. So I’m stuck with a hand held camera for now Sad smile

2000 bursting stars

- Rudimentary Sprite Batch – optimized and rendering 2000-4000+ sprites (stars) with some complex motion for each sprite. I easily got this painting up to 5000 sprites and with good frame rate. That’s simultaneously running sprites, you can create some pretty cool UX with that many independent sprites Smile

image

Icon Burst

- Sprite Batch + Sprite Sheet – This demo introduces the concept of a sprite sheet, a single sheet containing all your sprites that you cut up and draw on a surface. I wont go into the tool to create the sprite sheet, which is a work in progress, but the demo shows how to take that sprite sheet, push it to the GPU as a single texture and then render a clipped region of that texture via the sprite batch

image

Icon wall

– this takes the demo 2 above and renders a wall of icons . Very simple, I plan on doing more with this demo in the future.

image

Firework

– using SharpDx to get to Direct3D and Media Foundation this is taking the IE10 firework demo and doing it in XAML and DirectX. The result surprised me because it performed much better than the IE10 demo itself. I know its very rudimentary BUT it still does quite a lot of what the browser version does. Browser version was very choppy and was getting into the 20%+ CPU utilization, almost unplayable in some cases. The SharpDx/DirectX/XAML/Media Foundation version was very smooth!

image

Drop shadow Effect

– a simple Direct2D effect graph to draw a drop shadow on an image

image

Tilt-shift like Effect

– using only direct 2D supplied effects, several graphed effects to produce a tilt-shift like effect. Just shows that the supplied basic effects in Direct 2D when graphed can achieve some very complex effects, no need to create your own custom effect Smile

image

Tile Dashboard Editor

– A simple tile layout editor, I plan to use this to help layout my home screen tiles. Close to 80% of the apps in the win 8 app store use some form of tiled home screen, this tile editor will hopefully one day make creating these home pages very simple. The goal is to create a Windows Phone 8 like home screen layout of tiles.

image

3-Axis Grid

– this is a very basic introduction to the x,y,z axis in Direct3D and using matrix transforms to move the planes around. It also introduces the concept of Line-List to draw the lines, as opposed to the triangles I’ve been using to date.

image

Radial Control

– This is a very simple version of the OneNote Radial dial, the purpose is to introduce the concept of a curved menu and to show that drawing arcs in XAML is easy even without the Drawing.Shapes library that we are accustomed to from Blend Silverlight . This is all XAML and is very fluid. I plan to iterate this control many times, and hopefully one day it becomes a control I’d be happy to use in an app.

image

Demo Code

[please note that this will be updated when I have time to, I am quite busy and this is a side project. ]

image

image

Conclusion

Again please remember that these demo’s are the result of me learning these new technologies, they are very rough and I would expect you to re-architect them if you were to use it in your app.

I’m loving this new WinRT XAML SharpDx world and its pretty clear to me that MS are headed in the right direction with Windows 8.

I look forward to seeing what you build with this new application developer stack Smile