November 30, 2006 Minutes

Nov. 30, 2006

Owen Astrachan, Pakis Bessias, John Board, Shailesh Chandrasekharan, Tammy Closs, Ken Hirsh for Dick Danner, Brian Eder, Kevin Smith for Nevin Fouts,  Tracy Futhey, Christopher Gelpi, Susan Gerbeth-Jones, David H. Sanford for Guven Guzeldere, Billy Herndon, Bob Newlin for David Jamieson-Drake, Julian Lombardi, Roger Loyd, Dmitriy Morozov, Dan Murphy, Tim Bounds for Caroline Nisbet, Lynne O’Brien, Mark Phillips, Mike Pickett, Rafael Rodriguez, Molly Tamarkin, Robert Wolpert, Steve Woody

Guests: Chris Cramer, Klara Jelinkova, Bob Price, Marilyn Lombardi, Rob Carter, Ginny Cake and Kevin Miller, all from OIT

Start time : 4:05

I. Review of Minutes and Announcements:

• John Board – After our previous talk about spam, I was interested in whether we were seeing a one-time increase, or a continuing increase over time. We did some research in my department and found that we were seeing an 8 percent per week increase in mail volume hitting our server. Various of you are trying to get similar information on your systems and wondering what you should count, but it doesn’t matter what you count as long as you keep counting the same thing. This increase is causing unscheduled, unbudgeted expenses in improvements.
  Question -- Can we determine the cost to Duke of each non-spam message?
  Tracy – We have not created our own market economy to determine that. That could help us inform and wade through policy issues.

• Kevin Miller – In October I mentioned that our Time-Warner home broadband rates and services were changing for the better. New rates take effect Dec. 1. For the higher-end tiers, it’s about $15 per month decrease. For the base subscribers, the upload service is improving with no change in cost.
  The other thing is, while about 900 users are departmentally billed, any affiliate can get that at home. If they currently subscribe, it’s pretty easy, and it means a speed bump with no change in price.

• Tracy – We’ve just finalized Duke’s participation in an IT leadership program. It’s the second time we’ll be participating. They’ve had six or seven cohorts go through with our peer institutions. We were in the first round; we’ll be in next round in February.
  Those going include Lisa Berry-Setliff from the DHTS help desk, Ed Gomes from Library Services, and Kevin Davis, Kevin Miller, Jeremy Sisk and Kevin Witte, all from OIT. Each university usually sends six people. We’ll be sending a seventh, Scooter Freeney, who handles HR for OIT. He’s being sent because we want him to have a good understanding of what this HR leadership program is about so he can help us figure out who might be a good candidate in the future.

•  

II. Introduction and early observation from Klara Jelinkova

I’ve been here 14 days. I came here from the University of Wisconsin-Madison. My last position was assistant director for Systems Engineering and senior strategist for Research Computing. I worked mainly with large ERP implementations.

My early observation is that Duke is much more dynamic and spry than UMW. I would also say it’s achieving excellence on a greater scale than at a state institution. It’s really exciting to be here. I loved North Carolina as soon as I started to drive down here because it was snowing in Wisconsin when I left.

John Board – How do you see your job evolving?

Klara – I’m the director of Computing Systems. I have the privilege to work with some extremely talented people and the larger OIT community. I work with technical infrastructure services, server infrastructure, email infrastructure, ID management infrastructure. I love infrastructure, and the reason is you get to work with everyone and you get to form partnerships with people and I’m looking forward to it.

III. Update on spam statistics and email

Klara – I prepared a graph of the email system. In the inbound mail, we now get it in two ways, the old MX routing, and that is the piece we are getting away from. The Health System is one of the parts still on it.
It’s all very scalable. All of these machines are standard Dell pizza boxes and they can be replicated.
As the email flows it can go to a deptmental server or our post office, which is a vertically scaled Solaris server. Then in the outbound mail we have the SMTP outbound mail from departmental servers and from our mailboxes.
Then have our list servers and WebMail.

The issue we’re working on right now is, when we decided to have this vertically scaled large Solaris box with an external disc array, that is a somewhat limited setup because eventually you fill up that space. Also, when we were laying this system out, we did not realize there was the concept of metadata in our post office. The system was too slow for the metadata.
We have bought additional discs to hold the metadata and other data. We are doing that this weekend.
I just want to make you aware of the fact that that puts us about halfway through our redesign of the email system.
We have to address the outbound mail. We have to do a hardware refresh and eventually a redesign.
One question we should be asking is, should be we scanning not just inbound but outbound mail? We are going to replicate the same infrastructure we did with inbound – horizontally scalable that can be increased easily.

We will have to complete the migration to Sympa for listservs. We may have to look at all the lists to see if they’re being used.

As far as spam statistics go, when the new front edge devices were deployed we started to actually throw away some mail. We are using Sophos, which uses a probability ranking – more like a confidence ranking. If it’s ranked 96 or higher, we are not processing that mail. This graph shows the mail we are throwing away. I think it’s very fair to say it’s all spam.
Tracy Futhey – We told them that last time.
Klara – Oh good. The green is the mail we’re processing. This is the front line. People still need the additional spam filtering. Personally, I’m throwing away everything that’s 80 or higher.

Robert Wolpert – Do you have any sense of what volume this is? It looks like about half.
Tracy – It’s about half.

Rafael Rodriguez – Just FYI, I’m now at 25 percent or above and I’m not missing anything.

Shailesh Chandrasekharan – Is this filtering occurring in other departmental systems, or does it happen before?
Klara – It’s happening on the front end.
Tracy – Email comes to an address that’s duke.edu, even if it goes to your department, does it get filtered?

Klara - No, but you could work with us to set that up.

Chris Cramer – You’re mentioned that you’re seeing an 8 percent per week increase, at the old gateways we were about 6 percent. A lot of this on the back end is being insulated. It’s a question of what’s on the gateways. What does that do to the CPUs, etc.
As Robin was noting, if we have a doubling in spam hitting the gateways, that doesn’t mean we need a doubling of gateways. But if we are seeing a 6 to 8 percent rise (and it’s hard to get numbers because environment keeps changing), there’s going to have to be some kind of change. It’s not clear what it’s going to be.
There is SPF, and that mostly ensures that whatever the “from” address is, it ensures that address exists. It also depends on having a degree of saturation of all Internet sites – I think 25 to 50 percent. There also could be certificate-based email.

Others have mentioned OCR spam, which addresses jpegs that spammers use. These spammers are changing those jpegs so they’re not susceptible to OCR. That’s happened in the last three weeks.
Robert Wolpert – A wonderful ecosystem.
John Board – It’s a classic arms race.
Chris – We’re doing some things. If we see a bunch of mail from the same site, we can block those at the network level temporarily.
People whose home computers are turned off or who are not in the office over Thanksgiving. These numbers that John presented earlier, with the drop over Thanksgiving, I could believe that they’re based on availability of bots.

Robert – How high a fraction of its potential are we using of the CPU resources?
Rob Carter – At the moment it’s about 15 to 20 percent utilization. My expectation is we should be comfortable, provided we don’t see a fast rise or some change in the nature of spam.

Chris – These spam numbers are increasing exponentially.

Klara – We need to have a continued open dialog because these numbers are unknown. We are always able to add more hardware at the post office. At some point it’s a question of what is the institution willing to bear. We can either do more with hardware or tighten that 96 percent level.

Chris – It’s interesting because there are things we could do. There are appliance boxes we could install that will filter email. Another is to start working closely to identify what  the botnets are. We’d have to work closely with other schools and Time-Warner. I’m not advocating anything.

Molly Tamarkin – When is the outbound update planned.

Klara – I think we’ll start working on this in January.

John – Just as an aside, I’ve had students ask them to stop sending class information by email because they don’t want to wade through the spam, they want it on a website. They use IM.

IV. Interactive Technology Service Update

Julian Lombardi – As you know from our previous announcement, we have undergone a consideration of an optimal structure for academic services. We identified several areas that were in academic services, those are Student Technology Services, Digital Media Solutions, and two more groups – research computing that’s still undeveloped – and Interactive Technology Services is now being looked at. We’re looking for a director.
ITS deals with the intake of work from academic community and beyond that involves the development of interactive technologies.
Essentially, if you wanted a website built, you might go to ITS and they would provide you with the construction of the interactive parts of that site. They would work with Internet Framework Services, which works with the development. It’s sort of Web development, but the Web is not everything. It also involves other interactive technologies that might not be Web-related. So its name might be something of a misnomer and possibly confusing.
If you’re looking at Web services, these would be front-end services and back-end services.

Robert Wolpert – Does that mean php on one hand and Dreamweaver on the other hand.
Julian – Yes.
We’re looking for a senior manager of this group to serve as a manager of this and an elearning architect, with an eye toward interoperability, scalability, etc., so we can gain the efficiencies gained from this person.

John – What is the funding model?
Julian – We have not worked out the funding model. It will likely be a cost-recovery model that will fund this. We’re hoping that cost-recovery model and the work will allow for the efficiencies to benefit the university.

Tracy – We’ve had this inherent competition in trying to deliver Web services, are we focused on Google appliances, or are we focused on the people who want to come to the office and have a website built. There’s been a tension. We found that we were somewhat successful, but not as much as we could be.
Julian – That group might not build a website. A significant portion of that might be outsourced, but that outsourcing would be managed by this group to assure interoperability.

Q -- Will this ever extend beyond this to things like virtual environment?
Julian --  To devp anything, would have to follow the mission of the university and would need to fully understand the technology.
Would it include telephone interactivity, mobile devices? Absolutely. In that way it would support many of the other efforts we have under way.

Robert – About the research computing part, have you been talking to faculty? I would encourage that.
Julian – We’re hoping to start working on that in earnest in January or February.
Robert – I would urge you to not go too far before you involve faculty.

Bob Price – Just to let you know, we have some strong candidates for the ITS manager, so I’m confident that in the not too far future, we’ll have someone new to introduce.

V. Information Technology Risk Assessment – Mike Pickett and Marilyn Lombardi

Mike – Marilyn and I were asked by Tracy to help her with a university-wide effort. Internal Audit was trying to get the administrative units together to find places where we’re vulnerable – reputational, financial, etc.
The big issue we were challenged with was to identify things that were substantial. Not little things like where’s my USB key. Big things like does your USB key have Social Security numbers or student data.
So we pulled together part of the extended staff meeting, and brainstormed for ideas on pockets of risk.
Marilyn – It was a valuable exercise and I think what happened next was interesting because we had to pare it down to categories.

Mike – After this long list was created, we talked to some specific IT managers for people delivering services. We wanted written lists. Under each identified risk, we asked them to tell us what impact it will have on the university, what’s the probability that it will occur, and what might help us mitigate the results. That also brought a lot of volume of data.

Marilyn and I came up with three substantial areas:
1) information that was sensitive or confidential would be revealed.
2) safety, security and regulatory compliance
3) risks to our mission, our ability to deliver services.

For the first one, we came up with three issues.
1) The distributed nature of Duke kept coming up. We have strong processes in some divisions or schools. Information is very distributed and it’s possible for people to have information that’s on a laptop or outside server, etc., that’s sensitive and gets picked up by people outside the university.
2) Data is driven to more mobile devices. That has to cross the network or you could lose your PDA or USB key and suddenly the information is out there.
3) We also might get a new virus or worm that could get in there.
4) The possibility of a surplus computer leaving Duke with sensitive information on the hard drive. This is mitigated by processes we already have in place.

For safety, security and regulatory compliance, we’re seeing new laws and regulations almost daily. They’re putting more and more expectations on the university in general, but a lot goes through the IT shops. Also, as we’re getting more grants, agencies are having higher expectations on PIs and institutions for using and keeping information.

As for risks to our mission, we identified a failure of SAP, Blackboard, phones, etc. – failure of an enterprise-level technology, failure of a service to the outside world (www.duke.edu). We actually had an outage recently, but it was short. There’s risks in Telcom or somewhere else, like the lack of conditioned machine room space, problems with physical plants. Also there’s a large amount of fiber and copper cable running through Telcom. We also have staffing risks – do we have adequate staffing backup if people leave suddenly?

When we sorted that out, we decided the risks weren’t all that high. We’re still lower in the quadrant, where they’re not as probable and the impact isn’t as high. We would have to have a multimillion dollar failure to get on the scale. The things that would put us there are things we’ve already addressed.

Athletics, illegal drugs, student homicide, those things were all higher. Internal Audit has been helping the university think through this stuff.

IT isn’t an area where we have things that are really high risks. Things we have to watch.

Still, managers who helped said it was useful to go through this exercise.

Mark Phillips – Did we quantify not only real damage, but also reputational damage?
Mike – yes (shows slide of process of quantifying)

Rafael Rodriguez – How did you turn this graph into the heat map?
Mike – We didn’t, IA did.
Rafael – In DHTS, we had a different process.

Molly Tamarkin – The most damage I’ve seen working on systems that are a lot smaller have been due to human error, but I didn’t see that factored in here. Was that subsumed by another category?
Tracy Futhey – When major system goes down, we assume that could happen in any way, not just a system error.

John Board – What is the ultimate outcome of this document?
Tracy — The trustees are getting the heatmap from IT, compliance, etc., and all taken together they’re being lumped into a big heatmap with different entries (a dozen or a score) representing different areas. They’re talking about it in the trustees meeting tomorrow. Things in the upper right will involve intense analysis. Things that aren’t won’t get that intense scrutiny; I think that’ll be where we fall. I don’t think there will be any hardcore deliverables we’ll have to produce in the next two weeks.

End time : 5:30