SciRubyInterviews/ChrisHarrop

SciRuby authors Ara Howard and Justin Crawford interview Chris Harrop

Name

Chris Harrop

Title

Institution

Cooperative Institute for Research in Environmental Sciences, University of Colorado

Academic Background

B.S. Computer Science, Colorado State University, 1991

M.S. Computer Science, University of Oregon, 1996

Questions

[SciRuby] Can you briefly describe (a few paragraphs) your research project(s)? What are you trying to learn?

I am researching ways to maximize the reliability of real-time and retrospective weather and climate model runs on high performance computing systems (HPC systems). Today's HPC systems, particularly large clusters, have lots of hardware components that can fail: disks, fans, power supplies, nics, switches, etc. On top of that, a smoothly running HPC system depends on the flawless execution of a lot of system software such as the OS and batch systems. Simply put, a model which uses several hundred processors over a period of several hours or days is exposed to a high potential for failure. The management of a single model run is not very cumbersome for a typical user since it is easy to resubmit the run if it fails. However, it is common for experiment workflows to consist of hundreds or thousands of individual tasks with complex interdependencies. The difficutly in managing these complex workflows is compounded by unavoidable HPC system failures. I am attempting to build an automated workflow management engine for the weather and climate model domain to address these issues.

[SciRuby] Can you briefly describe (a few paragraphs) how computers are helping you learn that?

Computers are helping me by constantly surprising me with yet another failure mode.

[SciRuby] What effect would you hope your research would have on the world?

I hope that my work will help the weather and climate community run complex modeling experiments with enough reliability so that they can spend their time analyzing their results instead of baby sitting the running of their codes.

[SciRuby] How many people are working on your project(s)?

Just me, but I work with several scientists that have agreed to use prototype versions of my workflow managers.

[SciRuby] Are any of your team members computer scientists?

Just me.

[SciRuby] What's your computing setup, i.e., what OS, hardware, and language are you using?

I am working on two large Fedora Linux clusters. One is 768 nodes of dual 2.2GHz 32-bit Pentium 4 Xeons, and the other is 296 nodes of dual 3.2GHz 64-bit EM64Ts.

[SciRuby] What's your favourite software tool?

I don't think I have one, but if I had to pick I'd say Emacs

[SciRuby] Did you learn to program in school or on your own?

Both, but mostly on my own.

[SciRuby] How much of your work is programming?

Maybe 95%.

[SciRuby] Do you enjoy programming?

Yes.

[SciRuby] What's your favorite beer?

I have many favorites. Mountain Sun's Java Porter is great and I drink a lot of New Belgium Blue Paddle Pilsner. Basically anything that has a lot more flavor than carbonation is good for me.

[SciRuby] What languages have you used?

Basic, Pascal, C, C++, Fortran 77/90, Scheme, Lisp, Perl, Unix shells, TCL/Tk, HTML, Ruby, and probably more that I can't remember

[SciRuby] What is your favorite?

Ruby.

[SciRuby] Can you briefly describe (a few paragraphs) how Ruby is helping with your work?

Ruby helps me most by allowing extremely fast prototyping and testing.

[SciRuby] How would you define "scientific programming?"

I have sort of a liberal definition of scientific programming. I say that if the goal of your programming effort is to do or enable science, then you are doing scientific programming. If you write a program that inadvertantly enables science you are not doing scientific programming. You have to do it on purpose. In other words, writing a general purpose editor may help scientists, but it isn't scientific programming.

[SciRuby] Which kinds of coding problems do you end up spending the most effort on?

I don't know if this answers the question you were asking, but I spend a lot of effort in situations where I have no choice but to write a Unix shell script. Wrestling with bash or csh or awk can take considerable effort.

[SciRuby] Which kinds of coding problems do you end up spending the most time on?

See above

[SciRuby] Which kinds of coding problems form the largest obstacles for your field?

The largest obstacles are political. When a coding project involves multiple institutions, and one or more of them have arcane and/or stifling coding policies, one is often forced into using sub-optimal tools and techniques. Also, moving targets can make things challenging. When the requirements are constantly changing underneath you, it's sometimes hard to keep up. Fortunately, Ruby makes it easier to write code that can adapt quickly to such changes.

[SciRuby] How would you compare your software development practices to those of colleagues in your field?

I work primarily with physical scientists and meteorologists. For the most part, their software development practices are atrocious. But, in their defense, it's not really their field, and they want to spend their time doing science. I would say my biggest criticism is that they almost never insert any error checking code. Their scripts rarely return an exit status. They almost never bother to check if step A completed successfully before they go on to step B. I would say most scientists I've dealt with are focused entirely on getting the code to spit out the answer and pay virtually no attention to things like extensibility, maintainability, portability. The people I work with are often not willing to invest the time in developing for extensibility, maintainability, and portability because the benefit is long-term instead of short-term. If there is no immediate payoff, then it is viewed as a waste of time.

[SciRuby] What scientific achievement are you most proud of?

I'm proud of the fact that the prototype versions of my workflow management software have run hourly real-time weather forecasts at the Forecast Systems Laboratory with 99.6% reliabilty over the last year.

[SciRuby] What programming accomplishment are you most proud of?

I really hate to admit this, but I'm kind of proud of a really complex and weird piece of Korn Shell code I wrote that....Actually, I don't think I can even explain it. Suffice it to say that if I had a choice, I would have used Ruby. And if it were written in Ruby, it would not be very impressive at all. In fact, it would be downright boring.

[SciRuby] Tea or Coffee?

Coffee. Black.

[SciRuby] How do you test your code?

I don't use a formal method. Every time I write a method, I write a line or two of code to make sure it does what I expect. I test every possible scenario that I can think of.

[SciRuby] How much time do you spend debugging code?

Not that much. Since Ruby makes it so easy to thoroughly test everything as I go, I usually find and fix most of the problems immediately after I write a piece of code. I debug very small pieces of code at a time which makes the debugging process easier and faster. I do miss things, but testing every method right after I write it saves me a ton of time.

[SciRuby] If your project was going to be drawn by a famous comic book illustrator, what would it look like? (You don't have to answer this one; but if you do, your wish might come true)

[SciRuby] What do you do when you're not in the office/lab?

I drink beer and do Kung Fu. But not at the same time.