Title

Three Pitfalls in Java Performance Evaluation.

Abstract

Executing a Java application is a complex matter when looking at what happens under the hood. The virtual machine (VM) runs the operations and takes care of class loading, compiling and optimising code, and garbage collection. Due to the interaction going on between the VM and the application and the non-determinism (e.g., due to time-based sampling) within the VM, no two executions will ever behave exactly alike. Performance analysis in this context should not be underestimated. In this dissertation, we uncover three pitfalls that have not been taken into consideration prior to this research.

First, we show that one should not extrapolate performance results from one VM to another, and that small input sets do not necessarily yield behaviour that is representative for large(r) input sets. Second, we demonstrate that prevalent data analysis is falling short of the mark in many cases and can results in erroneous conclusions when making performance comparisons. We propose a rigorous statistical approach that deals with the problems posed by non-determinism. We also add rigour to one particular experimental design, namely, replay compilation.Finally, we illustrate that Java applications exhibit phase behaviour at the method level. We exploit this feature to allow a programmer to gain insight in the performance of his application by allowing him to locate bottlenecks and thus optimise his program by removing them.

Venue

As the examination is a public event, everybody can attend. I do ask that you drop a note, or leave a comment if you will be attending. Otherwise, you and several other people might find themselves without drinks at the reception afterward. The event takes place in the Jozef Plateauzaal of the Faculty of Engineering , J. Plateaustraat 22, 9000 Gent and it starts at 14:00.

If the university wanted to do something about the poor state of the building, we would have received new, double-glass windows, with high-insulating gas between the glass layers. But oh no … better we have sun screens.

I sometimes fail to grasp the decisions people at the upper echelons take. No, make that most of the time.

Give me a frakking break. (FYI, yeah, I admit being hooked on BG.)

Why would you want your users to go through the motions of logging in to the application portal (Athena), and from there start a browser which logs you in to a — right — web application?!!!?!?! The reason soon becomes apparent. You cannot do anything with it, unless you use IE. At least, neither Safari, nor Firefox or Camino seen to be able to show me anything I can use.

I am flabbergasted. Really. We claim to teach people good IT practices? I wonder where I can give feedback on this borked app.

Observer Effect and Measurement Bias in Performance Measurement

To evaluate an innovation in computer systems, performance analysts measure execution time or other metrics using one or more standard workloads (e.g., the SPEC benchmarks). The performance analyst may carefully minimize the amount of measurement instrumentation, control the context in which measurement takes place, and repeat each meas-urement multiple times. Finally, the performance analyst may use the appropriate statistical techniques to characterize the data. Unfortunately, even with such a responsible approach, the collected data may or may not be actually useful. In this talk we show how easy it is to produce poor (and thus misleading) data in computer systems research. We explore two common sources of poor-quality data. First, we get poor-quality data if our data collection perturbs the behavior of the system that we are measuring; this is often known as the “observer effect”. We show that even a seemingly insignificant measurement probe can dramatically alter system behavior; thus, perturbation is much more common than most performance analysts probably realize. Second, we get poor-quality data if we measure the system in a particular set of contexts and that set does not capture the range of reasonable contexts that a user of the system might encounter; this is known as “measurement bias”. We show that different contexts favor different configurations of the system. We conclude our talk by outlining techniques for producing high-quality data.

Dynamic Bug Detection for Managed Languages

Although managed languages preclude and help prevent some software errors, deployed programs still have errors and crash. In this talk, we discuss approaches for detecting bugs and making deployed software more reliable. Our work focuses on efficient on-line techniques. We overview approaches for detecting the source of null pointer exceptions and efficiently computing calling context. We present an approach for detecting data structures that are growing. We show how to piggyback on the garbage collector to summarize efficiently (in time and space) the object volumes and relationships based on their user defined class. Experimental results show this approach is effective at finding memory leaks, i.e., data structure errors of omission. We include a brief discussion of in progress work on tolerating leaks. These results indicate promise for inexpensive approaches that help developers find bugs and protect users from their consequences.

You can find a PDF of the announcement here. For your convenience, I have put together a map.

Here are the short biographies of the speakers.

Matthias Hauswirth

He is an assistant professor at the University of Lugano in Switzerland. He is interested in approaches for measuring, understan-ding, and improving the performance of modern, complex systems. He is particularly intrigued by the intricate interactions between the different system layers, from the hardware, over the operating system, virtual machine, application frameworks, all the way to the applications.

Kathryn McKinley

She is a professor at the University of Texas. Her research interests include compiler optimization, architecture, memory management, and software engineering. She is currently serving as the Editor-and-Chief of TOPLAS and has been the program chair of ASPLOS, PACT and PLDI. She has graduated eight PhD students and is currently supervising eight graduate students.

This is the first template I ever built in Pages, and I am much indebted to these nice guidlines on macworld.com. Of course, the template probably needs some polishing, so updated versions might become available in the near future.

• prof. Yolande Berbers, KULeuven
• prof. Kris Coolsaet, UGent
• prof. Koen De Bosschere (promotor), UGent
• prof. Lieven Eeckhout (co-promotor), UGent
• prof. Matthias Hauswirth, University of Lugano, Switzerland
• prof. Kathryn McKinley, University of Texas at Austin, USA
• prof. Jan Van Campenhout, UGent
• prof. Ronny Verhoeven (chair), UGent,

Basically, the internal defense amounts to a two-hour session in which I have to answer a barrage of questions from my jury. If I pass this test, I will be allowed to the public and second defense. April 11 is D-day for now.

There is, hoewever, a far less secure alternative. You can provide your password in a file, or interactively, to the kinit process which will then obtain a token on behalf of the application you wish to execute, by doing

kinit --password-file=filename application

Saving the password in a file (especially outside of AFS) seems like a major security issue, but that’s probably me.

Suppose you wish to run a job, and save its output to a file that resides on your home directory, which is, incidentally, and AFS mount. You log in, get a token (as per the manual method described here, fire up screen, start your job, leave screen (using the nice ctrl-a-d keystroke), and log out because you like to take your laptop home (which sounds almost like the purpose of having a laptop in the first place). The next day you arrive at the office, turn on the laptop, connect it to the omnipresent ethernet, and log into the machine (yes, you request a token again) and you check the output of you job … Not a good idea if you have a faint heart, or that data was really, really due, like yesterday.
Why, the innocent reader now asks? Well, clearly, because the output has not made its way to the file as you would have liked it to do. To cut short the story: the moment you logged out, your AFS token was revokes, or became invalid, or whatever, and the file that was greedily accepting your bits suddenly was inaccessible to your process. For example,

#!/bin/bash

i=0;
while(true); do
        echo "hup $i"
        sleep 1;
        echo "pup $i";
        i=`expr $i + 1`;
done;

woud yield this:

hup 0
pup 0
hup 1
pup 1
hup 2
pup 2
hup 3
pup 3
hup 4
hup 17
pup 17
hup 18
pup 18
hup 19
pup 19
hup 20
pup 20
hup 21
pup 21
hup 22

if you had logged out after 4 seconds and logged back in (and acquired a token) about 13 seconds later.

Boooooo! Eat that.

Of course, a simple workaround exists: save the data temporarily to a local file on the machine, e.g., in /tmp (and pray that the tmp dir is not wiped nightly). But that does kind of defeat the purpose of having a secure filesystem on which to store your data, does it not?