I found one reasonably decent shop where they sold breakfast, there are a few restaurants that might – might – serve decent food, and one Coyote Ugly saloon. I doubt one can have a quiet dinner there, but for a salad after the OOPSLA reception it might do

Java Performance Evaluation through Rigorous Replay Compilation, Andy Georges, and Lieven Eeckhout, Dries Buytaert

The abstract of this paper reads as follows.

A managed runtime environment, such as the Java virtual machine, is non-trivial to benchmark. Java performance is affected in various complex ways by the application and its input, as well as by the virtual machine (JIT optimizer, garbage collector, thread scheduler, etc.). In addition, non-determinism due to timer-based sampling for JIT optimization, thread scheduling, and various system effects further complicate the Java performance benchmarking process.

Replay compilation is a recently introduced Java performance analysis methodology that aims at controlling non-determinism to improve experimental repeatability. The key idea of replay compilation is to control the compilation load during experimentation by inducing a pre-recorded compilation plan at replay time. Replay compilation also enables teasing apart performance effects of the application versus the virtual machine.

This paper argues that in contrast to current practice which uses a single compilation plan at replay time, multiple compilation plans add statistical rigor to the replay compilation methodology. By doing so, replay compilation better accounts for the variability observed in compilation load across compilation plans. In addition, we propose matched-pair comparison for statistical data analysis. Matched-pair comparison considers the performance measurements per compilation plan before and after an innovation of interest as a pair, which enables limiting the number of compilation plans needed for accurate performance analysis compared to statistical analysis assuming unpaired measurements.

The bulk of this paper made up Chapter 5 in my PhD dissertation, which was published on April 30, 2008. Here and there slight improvements were made before we submitted the final version. You can get a preprint version of the paper. The presentation I gave is available in both pdf format or as a zipped Keynote archive.

The premise of her talk was the fact that there is a lot of information sitting idly around us. This information is not easily accesible. Moreover, relevant information that might help us understand our environment, or help us make decisions is rarely looked up when it is actually required. A typical example of this is shopping. There is much more we could learn, if the information about products was made available in a timely and intuitive fashion. Making this happen in a seamless manner is the goal of Pattie’s current work.

An initial step to making this happen is by equipping users with a wristband containing an RFID-reader. The RFID tag of various products can then be sent to a device capable of interacting with the information world, such as a PDA that can access the web.

This basic piece of equipment can be augmented by a gesture detection system. Such a system allows the users to rapidly browse the information, without requiring him to do awkward stuff of the PDA. A flick of the wrist might mean we want more information. The idea got a little scary when this is applied to people. Touching a person, e.g., by shaking his hand, could give you access to the public (ahem) information on that person. This would be handy when interviewing job candidates. It seems to me that there are much more scary possibilities, that invade our rights or privacy. I can imagine the police force finding this bit of technology quite handy when they have to hunt down people in a crowd. Legitimate use would be chasing criminals, but as we have seen in recent developments, people can be placed on a list of criminals (i.e., the no-fly list) for no other reasons than eating certain types of food.

Anyhow, the obvious step that follows the previous system is providing physical objects with an interface, i.e., creating meta-objects that can be attributed to real-world stuff. These object then provide the information people are interested in for each individual object.

Taking this further, makes us arrive at gaze-based interaction. Here, infra-red detectors determine the point at which somebody looks. A system like this can be used in musea, to broker information about specific details of a piece of art to visitors. Or it can be used to educate users about products.

Minimising the wristband yields a ring that can be used to point at objects. The target can then respond, e.g., by turning a LED green if the product matches your profile, or red if it does not. Said profilecould be a shopping list. Or a list with substances the user reponds allergic to. In this way he can select the goods that do not contain any of these substances without having to scan the list of ingredients over and over. Personally, I think this might be quite handy, although after several times shopping, you do get to know what you can buy and what not.

All of the above imposes technical challenges for both software and hardware:

• what is the user focusing on
• find relevant (personalised) information
• offer this information to the user in an unobtrusive way

I am not sure if this counts as unobtrusive but an example Pattie gave was the following. Objects could gain awareness, and try to attract your attention by lighting up when one passes by. An example is browsing a bookstore, where you can easily see which books match your interest.

A pretty cool example was that of the smart stickies. Current stickes cannot interact with people looking at it. Quickies are post-it notes that can be searched and located and send the writer reminders. For example, a to-do, a tag in a book, meeting data, etc. They can look up people in an address book an notify them automagically. They can also be used to keep track of personal items.

New research focuses on items that have I/O capabilities: I/O objects. An example is a pillow that can act as an interface to send messages to its peer, residing in your home. Another example was given by shutters that will automagically close to keep you out of the sunlight, while allowing light to pass to the rest of the room, so you can read in peace, without all that annoying sunlight getting into your eyes.

The most nifty example of an I/O object were the siftables. These are tiny machines, with their own screen, CPU, etc. that can detect neighbours. They can interact with each other when touching or residing near to each other. For example, if the screens show a face, the image is automatically adjusted such that the face looks at the neighbouring siftable. A possible use would be to teach people the concepts of OO-programming (which seemed to be the only relation the talk had with the main OOPSLA topics), as they have physical objects that can be manipulated and that respond to interaction with other object.

Pattie closed with mentioning the overall goal is to rethink how people can interact with the physical world by delivering just-in-time information in a non-disruptive way. This information would be accessed by different means than what we currently use: a keyboard and a mouse (or substitues for that, such as
a touchscreen).

First of all, it is important to recognize, people still want to meet in person, despite all the technological advances made during the last years that really means we can talk to one another without leaving our home, or the comfort of our couch. So to create a bunch of passionate users, you should build on this desire and get them together, where they can talk, discuss, learn and teach. A good example she gave was that of a barcamp. Basically, the roadmap to instilling passion in your users is discovering the characteristics these people have in common.

Essentially, where there is passion, there is a user kicking ass. So, you want people to get good at things, because if you suck at something, there is no way you will be passionate about it. Counterexample 1: I still suck at Haskell, but I’m quite passionate about it. Counterexample 2: the same goes for me and Drupal. But that may just be me. Most likely it is. On the other hand, there are people encouraging me to be passionate about this. With this observation, she encouraged the audience to give users a hi-res experience, such that they could understand that if they passed the suckiness threshold, they could really rock. This means showing things to people as you see them, and transferring your passion. For example, a mountain climber will certainly look differently at a rock wall than Joe average, e.g., me.

The main issue is that often people are encouraged to be passionate about a tool, rather than at what the tool can help them accomplish. Nice example is the camera. What you want to do is to make kick-ass pictures. So get people to be better at using the tool, helps them reach this goal. But don’t mention this, just show them the cool stuff they can do, without emphasising the tool itself.

Kathy claimed that the first roadblock to get things rolling is the brain of the users. People operate at a subconsious level, not at a logical level, despite the fact that – so she said – many men like to believe otherwise. Furthermore, brain have built-in crap filters, meaning it’s important to get the things you want users to retain pass those filters. The mind however operates at a higher level, but people are rarely actually guided by their mind. For example, when they really want something, they will seek arguments why they should be allowed to get what they want, rather than listening to their (correct) mind that tries to tell them not to for some good reason (e.g., no money). Her reasoning was that the brain cares about either life-sustaining of life-threatening things, about joy, sadness, anger, love, scary things, new things, intruiging things, etc. However, the brain cares not one whit about code. The brain also cares about faces, so it is more desirable to people to talk in person to somebody, instead of having them interact with a machine. The bottom line is that use have to use every trick at your disposal to capture the attention of the user. For example, conversational language captures people attention better than formal language. I guess that’s one to keep in mind for my PhD dissertation.

Well, everybody think he’s above average, even when it is not true. So the path to succes is constructed by dangling the carrot in front of the users: allow them to have the idea how cool they will be when they grow above the sucking threshold. So, users need a clear path to get to the good stuff, see a compelling picture down the end of the road, and a relatively easy first step onto the path. So, users must not be dumbed down. Additionally, the path must be open ended. Every time there must be a step users can take to keep increasing their competence.

Because all computer applications are somehow agnostic at some level to what users think, it is important that users can knock on the door when they get confused. Kathy called this adding a WTF? button to an application. One of the problem is that users guides and FAQs are written for happy people, not for somebody about to smack the hardware (after all, hardware is the part you kick). So FAQs are written for users on the other side of the canyon of pain.

I really like her metaphores.

Getting users on the path to world domination is but the first step, you also have to keep them going. The best way to do this is to get them in the flow state, which is the state where people just need to do this one extra thing, one more compile, and it will work, etc. They are not aware of time passing, and face it, you probably also accomplished most when you’re there. There seems to be a precarious balance between knowledge, skill and challenge. To avoid slamming users back to reality, you should make the good things easy, and the wrong things hard to do.

I think that she illustrated her point quite good by using the game example. In a good game, there is a payoff when a player attains certain experience, e.g., a faster car, better weaponry, etc., but there are also tougher challanges to be faced to get even further.

She made me grin when she asked how we feel about users. RTFM? That’s usually how I think most of the time, after having things explained time and again. But obviously it is the wrong approach, and deep down, we all know it. This is why some online communities are much more succesful than others. Those that don’t bark nor bite but care about newbies can get them up to speed easiest and create a following that will (hopefully) repond the same way later on when they passed the suck threshold. I think a good example of this is #haskell on irc.freenode.net, where people have always been helpful for as long as I’ve known. This is one of the key issues Kathy touched upon: get the community to help out. People should be encouraged to answer politely, and answer instead of just asking questions. Even when something is answered wrongl, the community can pitch in and correct things. And because newbies recall their pain best, it is important they actively contribute.

Moreover, when a community can be converted into a tribe, passion may be ignited sooner. A tribe has special little things only the members are in on, and they notice each other. Tribe members id themselves with a special thing, and they’re proud to be part of it. Provide people with the means to show they are part of the tribe, e.g., bumper stickers, t-shirts etc. I guess I should be taking my Drupal t-shirt out to the dry-cleaner so I dare wear it again. As Kathy nicely puts it: insider information is trading currency, so give your users something to talk about. Apple users should know what she means, given the huge number of rumour sites.

Finally, appreciate the intelligence of your users. A crowd can reach concensus, but that’s probably not what you’re interested in. Individuals can contribute better if they get the chance. So make them think they rule individually.

I really liked her talk, it was well paced, and had very well thought-out illustrations on the slides. She gave quite a lot of things to talk and think about. Which I what I intend to do.

From the scenario, identify user interface elements that help the user execute the tasks required to reach his goal. For this, Jeff advocated using post-it notes, as they can be reordered on a board, and teared up when wrong.

Given the post-it notes create paper UI building blocks, such as labels, input fields, buttons, tables, etc. these should be sticked onto paper sheets, which represent the screens a user sees as he proceeds through his stack of tasks.

Final phase is the testing. Get a person to play the computer, a person to explain the scenario, and a few people choosing to pick the UI-elements. Finally, somebody should observe and jot down everything the testers have trouble with.

The next step is to repeat the above phases as much as needed until you are confident the user interface you designed meets its requirements.

In all, I think this can be a good approach, especially when a complicated user interface needs to be designed. However, I’m still partial to the paper cutting and sticking the components onto paper. It seems to me that this might be a little time-consuming. This is especially true when you need a lot of screens, and the problem you’re dealing with is fairly complex. Basically, I think an electronical counterpart to the paper based cut-and-stick idea might work faster. Additionally, it seems as if the proposed approach is very hungry on people resources, so it might cost quite a lot to have a user interface designed in this way. And the testing phase seems to be quite awkward. But I’m no expert whatsoever. And a well-designed interface can of course rapidly earn its development price back because it can appeal to people, causing them to buy/use your application.

The abstract to this abstract reads as follows.

Java performance is far from trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism due to Just-in-Time compilation/optimization, thread scheduling, etc., causes the execution time of a Java program to differ from run to run.

This poster advocates statistically rigorous data analysis when reporting Java performance. We advise to model non-determinism by computing confidence intervals. In addition, we show that prevalent data analysis approaches may lead to misleading or even incorrect conclusions. Although we focus on Java performance, the techniques can be readily applied to any managed runtime system.

Statistically Rigorous Java Performance Evaluation, Andy Georges, Dries Buytaert, and Lieven Eeckhout.

The abstract reads as follows.

Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as Just-In-Time (JIT) compilation and optimization in the virtual machine (VM) driven by timer-based method sampling, thread scheduling, garbage collection, and various system effects.

There exist a wide variety of Java performance evaluation methodologies used by researchers and benchmarkers. These methodologies differ from each other in a number of ways. Some report average performance over a number of runs of the same experiment; others report the best or second best performance observed; yet others report the worst. Some iterate the benchmark multiple times within a single VM invocation; others consider multiple VM invocations and iterate a single benchmark execution;
yet others consider multiple VM invocations and iterate the benchmark multiple times.

This paper shows that prevalent methodologies can be misleading, and can even lead to incorrect conclusions. The reason is that the data analysis is not statistically rigorous. In this paper, we present a survey of existing Java performance evaluation methodologies and discuss the importance of statistically rigorous data analysis for dealing with non-determinism. We advocate approaches to quantify startup as well as steady-state performance, and, in addition, we provide the JavaStats software to automatically obtain performance numbers in a rigorous manner. Although this paper focuses on Java performance evaluation, many of the issues addressed in this paper also apply to other programming languages and systems that build on a managed runtime system.

This paper took quite some work, especially in the experimentation-wise. While the initial reviews were very positive, they required us to perform several extra experiments. But in the end, it was worth the effort. You can get a preprint version.

So, 2 out of X at OOPSLA for us! Yay!