Well, it was a pretty good day. I didn't know if I'd get this posted, but here we are. I have overhauled the Asset System based on customer feedback.

Assets are now driven from a database and tied to a license system. While many assets (images, videos, shapes) will be either proprietary and belong to the organization using the InfoMesa, some will be purchased and will involve licenses that must be tracked. Who's using the image "abc.jpg"? How many people are using "abc.jpg". These can be important questions - especially from a compliance perspective.

There are many uses for Assets. In the image above you can see images (from Assets) and XAML shapes (the ovals in this image). Images can be sourced from Hard Drive Farms (Network Shares, Local Drives, and URL's) -- this means that items secured in a Sharepoint Document Library can now be added as Assets for use in InfoMesa. Cool.

As you might remember, XAML is the underlying XML language used to describe shapes drawn and composited in WPF. I used Microsoft Blend to design the Ovals and the Integrin Shapes that could be useful for applications such as Cell Signaling and Metabolic Pathways. Since XAML supports animation, these things can also be animated - they can also be somewhat interactive.

The Assets Tab may look the same, but it's completely new underneath the hood. These Assets can also support keywords, so you can search for items in libraries by keywords and terms that may not be obvious from the filenames. This is a key feature in any non-textual repository.

Along with Assets, we made progress on Associations. Now this is something we have not talked a lot about. Associations allow you to tie elements together. They are not Annotations which are used to explain something. Association show relationship, they show flow (unidirectional or bi-directional). They are stored in the database and can be traversed along their paths. I decided to use curves for the rendering of the associations.

Each Association will have great control over their appearance. The Control Rods that control their shape will conceal themselves later on. They even have arrowheads to denote direction.

Even though this diagram is using XAML elements between the Association lines exclusively. I want to stress that *any* element can be associated to any other element. Web Results can be associated to an Image, a Video, a Molecular Model, etc.

Since these are not Annotations, any movement of the elements tied together will impact the lines. So feel free to tie stuff together and then move it - the lines will follow along.

Needless to say, there will need to be an option (most likely in the Settings Tab) to turn this complexity on and off.

Here's a close up of the Association Curve...

More next week...

Don't forget to vote if you live in the US!

I've been working this week on cleaning up the experience around selecting Rooms and Creating Rooms and Destroying them.

Here's where I am right now...

Now, a large profile consumes the top third of the screen. After a selection, the Name, Description, Creator and CreateDate can be readily seen.

Clicking on the Large Room Picture will bring up the Profile Dialog and allow you to add/subtract researchers to/from the room.

I'm also doing a huge makeover on the asset system, but that post will need to wait...I'm hoping to make the drop tomorrow night, but don't hold your breath - perhaps early next week at the latest...

InfoMesa is all about visualizing, persisting and collaborating around visual elements stored in virtual Whiteboards. Originally, we were just going to use Whiteboards as the base organizational element, but after some discussions with people in the pharmaceutical/Life Science industry we decided to organize these whiteboards further into the concept of a room. This will allow visualizations at higher levels of science occurring in the whiteboard complex. You can imagine rooms being grouped into pavilions, floors and even buildings and complexes.

Rooms can contain any number of whiteboards and you can create any number of rooms. Researchers can share a room or they can just be private, just as a whiteboard can be worked on privately or jointly. The taxonomy of rooms (how you name them) is completely up to the implementing company. For example, you could use phases of a project (Basic, Hit to Lead, Lead to Hit, Phase I Clinical) or make the whole room a Project (MRK445). The names of rooms can be changed and the whiteboards can be moved between them, so try not to worry about that too much.

Clicking on New Room will allow the creation of a new container for whiteboards.

The Researchers Tab will allow you to select Researchers that are "authenticat-able" to InfoMesa Assets. This will likely involve the cloud and Active Directory (and yes, we are working on this). We want to allow researchers from separate locations and even organizations to work on Whiteboards together, while also securing the assets in appropriate ways. This is a critical goal to approaching research in this manner.

Once a Room is created, you can build out Whiteboards.

There are some additional operations that you can do with Whiteboards, such as Moving them to other Rooms, Killing them off (only the author can do this), Copying them to clone work quickly, and Merging work from two or more whiteboards. In a future post, we will talk all about these operations.

As we have mentioned before, everything that's going on visually is being stored and persisted to a database.

When you create a Room you are populating the Room Table (as seen below)...

Likewise, the creation of a whiteboards records the basic information needed to tie a whiteboard to a Room.

By storing items in the database this way, we can allow organizations to build Rooms, Whiteboards and even populate elements on Whiteboards very rapidly (even non-visually) by writing data to the appropriate tables by using SQL Server Integration Services or any other ETL tool. Even Biztalk could be used to run complex workflow among instruments and people.

A new build for December's Phase is being dropped into the SkyDrive account found here:

http://cid-794708049c7ae9c2.skydrive.live.com/browse.aspx/TechnologyDemonstrators/InfoMesa/InfoMesa%20-%20Phase%202%20|5Due%20in%20December%202008|6

As always, comments and discussion is invited...

I've had a number of folks ask me to publish the requirements that have been given/generated for the InfoMesa application.

Here they are...I will continue to add to this document as they come in or get more refined.

General Requirements

• A User begins by signing into the application. This sign in is tied to Active Directory and/or other identification/authentication capabilities such as LiveID. Password expiration and strength are up to the organization implementing InfoMesa for their purposes.
• Upon authentication, a use is presented with a display of existing Rooms that they can use to establish Whiteboards. Rooms are self-provisioned and there is no limit to how many can be created. Rooms have names and Authors. There is currently no hierarchy of rooms, but that could easily be changed.
• A user can invite other users to participate in their whiteboard based on a role - reviewer or contributor. Contributors can add and modify content. Reviewers can simply view.
• Upon a Room selection, the Whiteboards are shown that belong to that room. These whiteboards can be added, deleted, moved to another room or launched for edit or review.
• Only an Author of a room can destroy a whiteboard or a room that they created.

Persistence

• Any item in InfoMesa is not only shown visually, but persisted to a data store that backs it. This data store is a third normal form relational database and will be will documented. Rooms, Whiteboards, Users, Elements, Tags, and ElementTags can all be built up from a database perspective if needed (using SSIS- SQL Server Integration Services or BTS, Biztalk Server).
• This data store allows backup, replication and scale up by silicon (CPU, Memory, Hard Drive Speed)
• This data store could reside locally on premises, on laptop or workstation or in the cloud.
• By Phase 2 (December 2008) this Data Store will be SQL Server 2005 (2008 capable). For earlier phases it is using Access Jet Database.

Whiteboards

• Whiteboards support Spatial arrangements in a very flexible manner - just like a physical whiteboard. Any element can be placed, rotated, pasted in front or behind other elements.
• Whiteboards support a Chronological view of elements. When they were added to the whiteboard by type (people, annotation, assets, gene expression maps)
• Whiteboards support a Commonality view of elements. What other whiteboards exist (that I can see) that have elements similar to mine?
• Whiteboards are 20,000 by 5,000 pixels wide and high respectively.
• Whiteboards can be linked to other Whiteboards. This allows for partitioning of elements, work, methodology and allows a natural manner to control performance.
• Whiteboards can link to other whiteboards in the same room or across rooms.
• Whiteboard contents can be copied to other whiteboards.

Sub Canvases

• Sub Canvases are miniature whiteboards unto themselves and act as scalable, scrollable windows.
• Elements can be moved into and out of sub canvases.
• The sub canvases job is to support structure in a relatively unstructured whiteboard space.
• Sub canvases are typed: currently we support Ideas, Hypotheses, Experiments, Decisions, and Procedures.
• Sub Canvases can be tagged.
• Sub Canvases can be rewarded or penalized to show how well the idea, hypothesis or experiment is doing or how relevant it is.
• Sub Canvases can be named.

Elements

• All Elements have author and create datetime stamps.
• Elements can be resized (or not).
• Elements can be rotated (or not).
• Elements represent anything placed on a Whiteboard (Whiteboard Links, Annotation, Associations, Sub Canvases, VLOGs, Sticky Notes, Ink Notes, etc.)
• Elements all share similar capabilities at an atomic level: Deletion, Rotation, Bring to Front/Send to Back, Move forward, Move Backward, Promote to Another Whiteboard, and Tag)
• Elements are never physically destroyed (deleted) they are logically removed from view.
• Elements record basic tag information based on their type at creation. For example, an Image will store it's width, height and image path characteristics at insertion.
• Elements can be tagged for filtering and identification purposes.
• Elements can move into and out of sub canvases for grouping purposes
• Elements can be Locked or Unlocked by the Author to keep from tampering.
• Elements can be controlled by permissions to allow certain people to work with it, or even to see it.

Formations of Elements

• By right-clicking and drawing a box around elements you can use a menu to create formations of elements in orderly ways...
• Line Up - All the elements lineup top to bottom, left to right
• Grid - the Square Root of the number of the elements dictates a square and the elements are placed into an equidistant grid.
• Fan- the Elements are fanned out like a deck of cards
• Cascade - the elements are positioned diagonally top to bottom and left to right
• Random Stack - random positions of x and y are generated within the selection box and the elements are distributed to these points
• Tidy Stack - elements are stacked tightly and neatly top to bottom and left to right.
• Loose Stack - stack is nearly on top of itself and items are rotated slightly as if thrown into a loose pile.
• Magazine Stack - the stack is skewed and shows it's depth by edge.

Tags

• Tags are definable at preset ontologies or ad-hoc grouping mechanisms.
• Tags can be used to filter the items on a Whiteboard, either spatially, chronologically, or from a commonality perspective.
• Tags can have a hierarchy (single parent groupings).
• All meta data about an element is stored as a tag.

Annotations

• Annotations float above the surface of all elements.
• They are easily moved and arranged.
• Lines of many styles and varieties are spawned from these annotation points.
• VLOGs, Sticky Notes and Ink Notes can be connected to these annotation points.

Associations

• Elements can be associated with one another.
• Associations can be build by similar/shared tags.

Services

• Support for Web Services with branded experiences - this is for our partner community to build out...
• Web Search is persisted (which means that the web query can be re-processed for the user).
• Web Query/Search Results - each web result is shown as a thumbnail of the actual page, instead of a list or results. Double-clicking will launch a browser with the URL.

Tools
Currently we will ship these tools for Phase 2 (December)

• Link to other whiteboards
• Annotations
• Associations
• Sticky Note
• Ink Note
• VLOG
• Gene Expression Map
• Molecular/Protein Viewer
• 3D Model Viewer
• TileMap
• ScatterPlot

Shelf

• Send "Data-built" Elements to specific Whiteboards without knowledge of how it fits in the overall project - similar to an in-box.
• Instruments in a lab can send lab results directly to a user via web services (like Thermo-Fisher).

Assets

• Assets are images, geometry or data clouds (2d, 3d, telemetry, temporal) that can be placed on Whiteboards.
• Assets can be tracked from a licensing perspective to constrain miss use or to support direct targeting of groups.
• Assets also support XAML geometry (XML strings) that can represent shapes (even animated shapes) that can be positioned, rotated and layered.

Guided Tours

• A tour can be set up with way points (x, y and scale [magnification] of various points along a whiteboard) and audio recording.
• They are completely passive, not interactive.

Settings

• Ability to configure Data Sources to feed ScatterPlots, Tables, 3D Models, Browsers, and Gene Expression Maps
• Ability to turn association lines on and off.
• Ability to turn labels (time stamps and author stamps on and off).

Other Devices

• Inherent support for web cams to record research findings.
• Inherent support for large, connected multi-monitor setups
• Inherent support for ink and gestures
• Inherent support for Surface Computers
• Inherent support for Digital Tables (Wacom)

What is WPF?

Windows Presentation Foundation is the new visualization library that shipped with .Net 3. It is supported inherently on Vista, but also works with Windows XP (sp2). Like it's older (much older) cousin GDI, WPF is used to paint items to the screen. GDI has many flaws that WPF improves upon. For one, WPF is built on top of DirectX, this means that anything using WPF is being driven off of any graphics acceleration cards (GPUs) that happen to be on board. This type of off-loading has a major impact on both performance and what can be accomplished visually.

In GDI, anything painted on the screen is not persisted. In other words, for whatever is painted on the screen, the burden for keeping it on screen during refreshes and occlusion (minimize, windows moving over it) falls to the developer.  Additionally, GDI is basically a fire and forget operation. Pixels are turned on and off, but the ability to work with the pixel directly is not easy - it's not an object - it's just memory with a value. WPF is different, it's object -based, so a rectangle persists, has properties and even events and methods. It's very much like a Visual Object-Oriented Database.

I thought I'd use InfoMesa to show some of the benefits of WPF. First, when you use the geometry of WPF you get resolution independence and thus, scale independent displays. Even raster based objects, like Images, get a pick up. Here's an example. In the screen below we have a Metabolic pathway, made up of about 25 images that are pasted together - each tile is positioned precisely on a grouping mechanism called a Canvas.

The magic begins as we zoom in (in this case, we use the Mouse Scroll Wheel to move in)

Again, we keep moving in, the full image is still being handled, but now we see paper-crisp sharp detail.

XAML - understanding the markup and programmatic side

Positioning each item can either be done programmatically or by using XAML markup. In my opinion, the XAML markup side has been clearly described over the last few years, but the programmatic side is where all the power comes. If you are going to use WPF for visualization purposes you have to learn the programmatic side to gain all the advantages.

For example, to create the Ink Canvas below you could use XAML...

<Canvas>
<InkCanvas x:Name="inky" Width="100" Height="100" Background="LightGray"/>
</Canvas>

Or, I could do it programmatically (in C#)...

InkCanvas inky = new InkCanvas();
ParentCanvas.Children.Add(inky);
inky.Background = new SolidColorBrush(Colors.LightGray);
inky.Width=100;
inky.Height=100;

It's not an either/or street - mixing them together gives you tremendous power. In InfoMesa you can right click on any of the containers for the visuals and create a menu such as the one below.

Selecting the option that's second from the last (Dehydrate XAML) takes the object (that, in this case, was programmatically generated) and shows the XAML that represents it.

Notice the XML tag element called Strokes at the bottom - yeah, that's right, that's the actual ink strokes in the ink canvas. You can store this value and persist visual changes to the items (without needing the database that may have been used to build them out). This is a really big deal for Scientific Visualization.

Above, I show the Ink Canvas on the InfoMesa surface and also the Asset Gallery which can contain imagery or XAML based Markup. These Shapes (they can be different) can be stored as strings or XML proper and pulled out of a repository and squirted directly on the canvas.

More articles are on the way and a new build of InfoMesa is coming next week.

This is an exciting day. The project that I've been working on for almost a year now is now going into an official launch. InfoMesa has been, and continues to be, a community effort by Microsoft Life Science and our major Enterprise Life Science customers and partners. InfoMesa is a bit unusual because it's designed to create community in the scientific landscape, while also having broad applicability across many other vertical industries.

What is it?

InfoMesa is a project to allow scientists to do more science and more discovery in a collaborative and data-rich environment. The metaphor that we have elected to use as the underlying fabric of the InfoMesa is a Whiteboard.

InfoMesa allows any kind of data or visualization to be added to the Whiteboard. Far from static, these tools are interactive, allowing data to be absorbed from data sources like Oracle, SQL Server, Excel Spreadsheets, XML or even Cloud-based web services. InfoMesa, when complete will support imagery, video, 2D connected models, 3D models (lit in a photo realistically manner), web searches, results from web service calls, Image Tile Maps, ScatterPlots, Sticky Notes, Ink Notes, Rich Annotations and Associations.

An Example of a typical InfoMesa Whiteboard is shown here:

Whiteboards are quite large in size (about the size of 20, 30" conjoined monitors). Whiteboards can also be linked to each other, so this allows an efficient way to both organize work and to handle performance concerns. The screen is completely resolution independent, you can use the MouseWheel to zoom in and zoom out and the canvas can be scrolled left, right and up and down.

Each element on the Whiteboard can be tagged (and actually will insert tags based on it's type when a tool is added to the whiteboard).

Data Visualization backed by Data Persistence

As each item is added to the whiteboard, it is saved -- or persisted -- to a back end database. Each whiteboard is drawn from a database that contains a third-normal form relational data structure. This means that as you add items, other people that are sharing the same whiteboard can see what you have added and vice versa. It also allows us to control who can see, change, and create items on the whiteboard.

Another advantage to this is that a whiteboard can be built programmatically. In other words, by doing appropriate data inserts and updates in the data model a whiteboard can be built up.

The platform is currently sitting on top of an Access Jet database for easily being transferred to any Windows XP (sp2) or Vista machine. However, the final data platform will be SQL Server 2005 or 2008 for scalability, security, and feature reasons.

Another item to throw in here, Sharepoint Portal Server, which has been explosive in both it's growth and uptake in the Life Science Community has a feature called the Business Data Catalog (BDC). This feature allows Sharepoint to index relational databases and customize the experience of what happens when a search result link is clicked. This allows searching across the entire corpus of Whiteboard, Room, Element and Tagged Elements and when a result is clicked an InfoMesa application is fired up and presented to the user.

How are Things Organized?

To have science organized by massive whiteboards is too flexible, everything just gets grouped together spatially and every scientist or worker will have their own way to doing this. We need structure and it needs to be inherent to the underlying fabric.

So, we have come up with a hierarchy of organizational principles. Rooms are provisioned by a researcher, or by a team of researchers. Each room then contains and number of Whiteboards. Whiteboards can be linked to one another within a room metaphor. There are some use cases floating around which require links across rooms and we will be pursuing this.

Within each whiteboard, we support a notion of sub-canvases that are typed. We currently support Ideas, Hypotheses, Experiments, Decisions, and Procedures. This allows a structure to embed any of the tools inside these frameworks and move them around, persist them to another whiteboard, or even to Reward or Penalize the results coming from the type. For example, if an experiment seems to be promising a bar graph on the top left corner can be clicked and adjusted to signal that the experiment seems to always be successful given the conditions described inside the canvas.

The Problem we are Trying to Solve

Why are we doing this? During the last year I've had a number of scientists basically support the notion that research and the computing environment underlying this activity as it stands now does not support the learning, annotation and journaling (documentation) richness that is needed. As an example, consider the following web site: www.pdb.org - the Protein Databank. This is a fantastic site and used by hundreds of researchers every day.

This site provides an efficient way to store information and to share it, but the granularity of the storage and the learning aspect is completely inefficient. I can see who the authors are, the primary citation, the history, the method, and even parameters to consider about the data. If I want to interact with the proteins structure I can launch any number of molecular viewers (KiNG, Jmol, etc.). Once I have read everything and learned about the protein, I close everything back up and the knowledge is now mine, but the company or organization that I work for is not any smarter (other than the knowledge between my ears now).

What if, the actual areas of the protein structure that Dr. Zhang worked on was clear to the eye, what if the specific data points in the X-Ray Diffraction data that are critical to understanding the structure or mechanisms in the structure were able to be annotated and connected literature? This is what we are solving with InfoMesa. These types of pages can be sourced into InfoMesa, but the annotation features and manipulation features allow me to do the science inside of a rich cockpit, without worrying about the communication aspects as much. In short, I can do more science and the knowledge is institutionalized, regardless of geographic boundary.

Will it work on my Laptop?

When it comes to an application like InfoMesa we are trying to revolutionize the actual "Day in the Life" of a researcher. There are now numerous form factors that both Microsoft and others are bringing to market that allow workers to be more social and more productive. Examples of this are the Surface Computer, Touch Wall, Social Computing Rooms, such as those found at Renci, and even powerful desktop machines with 90" of conjoined monitors such as those sold by Alienware, HP Blackbird and Boxx Technologies.

We do support InfoMesa on Laptops and I have been demonstrating that for over a year. It is not the platform we expect that it would be used in typical use.

Additionally we are exploring how it would work with technologies like the Wacom Cintiq digital tablet and other form factors.

What do I need to run it?

Microsoft Vista Recommended (Also supported by Windows XP, Service Pack 2 - DotNet 3.51)

Visual Studio 2008

A *great* graphics card

Instructions: Unzip InfoMesa_Data in the root drive of C: So the structure should look like this:

c:\infomesa_data\3ds
................\assets
................\Database
................\pdb
................\tilemap

Unzip the InfomesaLoader in the Visual Studio 2008 Projects Folders (found under your Documents folder)

Where can I get it?

InfoMesa is an ongoing project. We will be publishing the community members soon on these pages, but anyone can download it and play with it. We suspect that even the community users will both share and "fork" the source code to do other projects internally that they want to keep proprietary.

http://cid-794708049c7ae9c2.skydrive.live.com/browse.aspx/TechnologyDemonstrators/InfoMesa/InfoMesa%20Phase%201%20|5Due%20in%20September|6

This link above contains both the source, the data and example files and a PPT to share with management or others.

To Be Continued...

I will continue to post here on a weekly basis regarding the progress of this project...send me comments or questions that you have.