Thimbles & Care

Culture Hacking, Reloaded

University culture is basically feudalism. (It started, after all, in the middle ages.) That’s likely non-optimal.

Apparently the problem that I blogged about last week where Knight Capital lost $440 million in 45 minutes was not caused by a software bug. Instead, it looks like the program written to generate fake transactions while testing in the lab was accidentally included in the package when the trading program was moved to a live test. I wouldn’t want to be the person who put together that package.

While IBM’s John Ward was upgrading the memory on our z10 BC Sunday morning, I took some pictures.

The front of the Central Processor Complex drawer on our z10 BC

Starting from the left, there are power supplies, then two service processors (little computers that manage the configuration and microcode and such), then six slots for I/O fanouts (only two are populated, since we only have two I/O cages and no coupling links), and then two timing circuits.

Top view of the CPC drawer with all memory removed

Here is the top of the drawer after John removed all the old memory cards. In the front you see the top of the heat sinks for the processor and controller chips. The four chips on the left and right are the processor chips while the two in the middle are system controller chips (they manage the clocks and memory accesses and contain L2 cache.) To the left are the tops of the power supplies. At the back are the empty slots for the memory cards.

The new memory cards after installation

Here’s the back of the drawer after John has finished installing the new memory.

Like anyone else who’s ever written a program, I’ve written programs with bugs in them. But I’ve never aspired to writing a program that can lose $10 million a minute.  “Buy high and sell low” doesn’t sound like a path to fortune, although apparently it can get you some fame.

I was thinking about viewing SPOOL files, and I thought I’d share the story of how *UTQA came to be written.

COM-PLETE comes with a SPOOL utility, *UQ. Some time around 1990 or so, we got a new release of COM-PLETE that significantly changed the appearance of the output of the A subcommand, the one that shows active jobs. Lots and lots of developers were complaining, with some reason since the old version had showed drained and idle initiators while the new one doesn’t. We had recently installed the first version of Natural Process (since renamed Entire System Server) and I was figuring out what could be done with it. As I read about the SPOOL-related views, I thought, “I could write a simple program that replicated the old behavior of *UQ’s A screen.” So I did. I gave it what I thought was the obvious name, *UTQA, since it was a UT utility that replicated the old *UQ A subcommand. I showed it to John Camden and he wanted it expanded to replicate all the functionality of *UQ. I didn’t see much point in that, since everything else in *UQ still worked like before. However, Marshall Thomason, our DBA at the time, decided to run with it, so he wrote the rest of this application. A few other people have maintained it since then; in particular, Jim Bullock added support for sending output to “green print” a few years ago.

So that’s the story. If you’re wondering what I use, I usually use Natural ISPF. I’ll go to *UTQA when I want to see idle or drained initiators, or to use the green print function. I still use *UQ to scan SYSLOG, or for its non-SPOOL functionality.

The past few days we’ve had folks from the Kuali Foundation here giving us an overview of their projects, so I thought I’d post my reactions.

I think the Kuali business model—developing open source software in a consortium of similar organizations—makes the most sense for the University. Our current model of developing everything on our own seems less and less sustainable as time goes by; even if we were to continue developing what we’ve already done indefinitely we’d want to try to find other partner institutions to share development efforts with eventually. I think it would be easier and cheaper to convert to something open source like Kuali rather than a vendor product where we would have to rely on their documentation rather than being able to look at the source code to see what’s actually going on. Also, as a member of a consortium we would have more influence on product directions and features than as mere customers of a vendor. The only real advantage of going with a vendor from a management perspective is you have someone else to blame when things go wrong.

Given that the business model seems good, the only question is how is the quality of the product. Now, it’s hard to judge that just on presentations and demos, but they seem to have done a reasonably good job of designing and implementing their systems. So if it were up to me, I’d say we should plan to convert to Kuali.

This is cool: a Turing machine built from a Lego Mindstorm Nxt kit.

A Turing machine is a device invented by Alan Turing to study what it means to “compute” something. (This was before electronic computers were invented.) This Saturday (June 23) is Alan Turing’s 100th birthday.

Just to provide some support for point 1 in my last post:

Cringely: A lesson on IT labor economics from Memphis

When ServiceMaster announced its decision to cancel its contract with IBM and to in-source a new IT team, the company had to find 200 solid IT people immediately. Memphis is a small community and there can’t be that many skilled IT workers there, right?  ServiceMaster held a job fair one Saturday and over 1000 people attended.  They talked to them all, invited the best back for second interviews, and two weeks later ServiceMaster had a new IT department.  The company is reportedly happy with the new department whose workers are probably more skilled and more experienced than the IBMers they are replacing.

Read the whole thing. No, really, read it.

Labor costs represent a very high percentage of IT support costs. Managers who note this usually think the solution is to find cheaper labor. This is invariably wrong.

I’d like to expand on Adam’s post on recruiting and retention, and also tie it in to our earlier policy to build our own administrative applications instead of buying commercial packages. This was the reasoning behind Lawrence and Randy and company managing Data Processing the way they did:

1. The key to any successful organization is the people who make it up. You should do whatever you can to recruit and retain the best people you can get.
2. The University can’t compete with private industry on salary. If we want to hire and keep good people, we have to provide something they can’t get in another job.

So what were the non-financial incentives they used?

• We recruit and hire people with little or no experience, but strong aptitude and people skills, and train them. In this way we get good people that other IT organizations miss.
• We provide interesting and challenging work. This is where the “build” strategy beats out “buy”: developing your own application requires more creativity than installing and maintaining something bought from a vendor.
• We show respect and trust by allowing by providing direction and resources, but allow individuals to work out their own creative solutions to problems.

I agree with Adam that it wasn’t just the creation of ITS that changed this. It started with the simultaneous dot-com boom and Y2K remediation. For one thing, outside salaries increased more quickly than the University could keep up. Your job needs to be really interesting to justify not leaving when doing so could mean doubling or tripling your salary. At the same time, the kind of remediation that was needed for the Y2K stuff wasn’t all that interesting. Then ITS was formed and we had to do SSN remediation and by the time that was over the culture (and people in management positions) had really changed.

Still, I wanted to point out that there was a well thought out, coherent philosophy and strategy behind the way Data Processing/ACS was managed, and that it worked quite well for many years.

Ars Technica has a good retrospective on HyperCard, a program I still remember fondly. (In fact, I still have my copy of HyperTalk 2.2: The Book in a bookshelf at home.) I wrote a JCL tutorial in HyperCard; UT developers who’ve taken my JCL class have seen some of the content that was in it.

When Tim Berners-Lee’s innovation finally became popular in the mid-1990s, HyperCard had already prepared a generation of developers who knew what Netscape was for.

Yep.