Members present: Ed Anapol, Mike Baptiste, John Board, Dick Danner represented by Ken Hirsh, Angel Dronsfield, Brian Eder, Nevin Fouts represented by Stephen Galla, Tracy Futhey, Patrick Halpin, Craig Henriquez, Billy Herndon, David Jamieson-Drake represented by Bob Newlin, David Jarmul, Eileen Kuo, Roger Loyd, Greg McCarthy, Melissa Mills, Caroline Nisbet represented by Kyle Johnson, George Oberlander, Lynne O'Brien, Mike Pickett, Rafael Rodriguez, Molly Tamarkin, Robert Wolpert, Steve Woody
Guests present: Chris Cramer, OIT; Bob Currier, OIT; Dan McCarriar, OIT; Michael Gettes, OIT; Rob Carter, OIT; Phil Lemmons, News Services; Ginny Cake, OIT
I. Review of minutes and announcements:
American Tobacco Warehouse
Tracy Futhey announces the university has signed a contract for space in the soon-to-be-renovated tobacco warehouse. Eventually much of OIT will be consolidated in the new space, although some functions such as the Help Desk will need to maintain a presence on campus. Completion of the renovated building is about a year out.
Content Management
Dan McCarriar announces a contract with Zope will be signed by end of day tomorrow. He is scheduling introductory meetings, and he will ask them about distributing the software.
Teaching Resources Web site
Mike Pickett announces the new Teaching Resources Web site (TRW). Eighteen departments participated in the design and content of the site that was put together by Jen Vizas and Cheryl Crupi. It is a great resource for teachers. The current design is only the first phase and is expected to change.
Social Security Memo
Tracy Futhey announces Dr. Trask will be sending memo telling departments and organizations to get away from using SSNs as identifiers or authentication mechanisms, and SSNs should only be stored for legitimate reasons, e.g., tax records. We need to make sure SSNs are not used in any systems in addition to not being used as identifiers.
Calendaring Futures Forum
Mike Pickett announces an upcoming calendar futures forum. It will be a shoot out between Meeting Maker, Lotus Notes, Groupwise, and CorporateTime. In April we laid out criteria about what we're looking for. Maybe we'll see what the future of calendaring software at Duke will look like.
II.Virus filtering updater
Rob Carter
Chronology: On August 1, OIT rolled out new mail gateways to give us more stable virus facilities. On the night of August 18, Sobig.f hit. By 11 A.M. OIT got an updated DAT file from McAfee and installed it on the gateways. By the August 24, 2.5 million copies of Sobig were thrown away before they could be delivered. This represents a 50% increase in daily e-mail traffic directly attributable to Sobig.
Also, there used to be a 5-7 second delay on messages through the gateways. The new gateways dropped the delay to 3 seconds. During the worst viral traffic the delay rose again to about 6 seconds. Now the viral explosion seems to be trailing off. Messages are also being checked for spam.
The point is that the new gateways have a lot of capacity. The current gateways handle about 2 million messages per day, fully analyzed. OIT is going to replace the old backend post offices with new equipment. The replacement is expected to handle even more traffic. If the front end hardware had not been replaced (i.e., new gateways), the backend would not have held up during the high viral traffic period.
Michael Gettes asks if we are able to add capacity and scale out horizontally.
Rob Carter says, yes, and it is almost linear scaling.
Robert Wolpert asks if other universities were hurt as bad as we were.
Rob Carter answers that many were hit worse.
Mike Pickett asks if someone is not on acpub but uses mail alias are they getting messages scanned too.
Rob Carter says anything addressed to @duke.edu goes through our gateway so yes they are getting scanned.
Ken Hirsch adds that the Law school updated the McAfee DAT files on their systems, but the scanning machine choked on the volume of traffic. The Law School cleared about 11,000 messages out, but around 75% of incoming mail to Duke Law was Sobig virus.
Melissa Mills asks if we send from one duke address to another, is that mail scanned.
Rob Carter answers yes, it should be.
III. Network threats - worm, port blocking, etc. update
Chris Cramer
Background: Four weeks ago while attending a conference in Athens, GA, Chris got a call from staff that several hundred computers at Duke had been hacked. This was the dCom vulnerability. IT Security found that seventy-five percent of computers on campus had not been patched even though a patch was available. Quickly, OIT blocked port 135 into campus because they realized someone was likely to release a worm in the near future, and staff also worked to get campus machines patched. The assumption is that before all computers were patched, someone brought a laptop from home, connected to the network, and the worm was released. OIT isolated the affected portion of the network, and that worked well to control the worm. Unfortunately, another worm, Natchi, was released and it was very effective in spreading. The traffic was such that the CPU load went to 100% on the routers and at that point the routers stop routing. There are now inbound and outbound blocks on ResNet because it is fairly well infected with Natchi. OIT is letting folks know if their machines are infected.
Overall, Chris thinks Duke has been handling these problems very well.
George Oberlander points out that all of these hacks, while they did paralyze the network, are rather benign compared to what they could have done.
Mike Baptiste adds that he is still chasing two or three worm machines every day. Users install new operating system on new machine and they are infected. He is looking into partitioning the Pratt School network. He thinks he could have done better if he had more control over the network inside Pratt.
Chris Cramer says one of the things to consider is that the larger the perimeter firewall is, the harder it is to protect.
Michael Gettes is concerned that when you start doing anything that closes down normal ports you're forcing people to find other means to get in. If you leave all ports open you can block the port that things are coming in on, but if you block all ports, you force attackers to tunnel in some other way. If you leave them open you can deal with the event when it happens.
George Oberlander feels if you go the other route and use a host-based personal firewall, you are dealing with an entirely new support model unless you have ability to centrally administer it.
Michael Gettes maintains that when you start blocking ports, port hopping will happen. Or Microsoft will start putting services on other ports. So by port blocking you end up with less control over the ports. Also, there are lots of types of machines on network that do things differently so one firewall breeds laziness. One thing out there does not protect us. 1000 machines should have 1000 firewalls.
Kyle Johnson says we need some way to patch machines faster.
Tracy Futhey says that will be a topic at the next extended staff meeting.
John Board asks if there is anything obvious we could have done better.
Chris Cramer says when we blocked port 135 that first week it might have helped if it had been implemented sooner. He did not realize the scope of the problem at first.
IV. Webmail service update
Rob Carter
On August 18 OIT rolled out a new IMP-based Web mail service. The old SilkyMail service is still available, but will go away in October. The new service is running on brand new hardware that is much faster than the previous hardware. Since the rollout, 153,618 sessions have occurred using the new Webmail service, originated by 13,022 unique users. That's about the same number as used the old Web mail. Changes made to front end apparently were effective in getting people to try the new service.
Also, department Web mail service was moved from DLUG (Duke Linux Users Group) servers to OIT servers, and no problems have been reported.
V. Personal firewall software update
Chris Cramer
Contrary to popular opinion, OIT never had a sight license agreement for ZoneAlarm personal firewall software. However, this year OIT purchased a license to distribute Kerio firewall software. It is available to any student, faculty, or staff who have a valid NetID. Kerio is more admin friendly than ZoneAlarm, but it is not a centrally administered firewall.
Ken Hirsh says Kerio does not play nice with the Duke VPN client.
Chris Cramer says that in general VPN does not like any firewall, but he does not have any Kerio-specific reports of that. However, there is a known issue with Dell Latitude laptops. Other laptops work fine. Other desktops work fine. Dell has been contacted about the problem and is working on a solution.
VI. Network monitoring/bandwidth, policy
Chris Cramer, Bob Currier
For the past two years Duke has seen a large amount of ResNet bandwidth leaving the university. As a temporary measure the outbound traffic on ResNet was limited to 100 MB in 2002. That limit was quite generous, relative to what we know about how bandwidth limits have been handled at other peer institutions. In principle, this should have worked. In reality, what happened was we inadvertently limited the inbound as well as the outbound traffic on ResNet.
At another university, a study was done where letters were sent to the top bandwidth users, most of whom did not know they were consuming so much. The question is, can we do something similar now? We met with the current DSG president and Eileen Kuo (student ITAC representative) and discussed ideas for addressing this issue.
We have the capability to determine a per-computer bandwidth usage. With that information we can limit bandwidth usage per computer. We talked about a 5-GB per day limit and about educational programs. We also talked about consequences for violators. Even if outbound traffic is limited, internal traffic (on campus) is not affected. We like the idea of sending a letter each time a daily violation occurs. After seven letters, you get speed limited. All we keep track of is number of letters sent. We don't keep incriminating data that can be used against the violating student because we delete that data every day.
Bob Courier says current data shows 10% of students are using 98% of the bandwidth. Or 90% of students are using 2%.
Mike Baptiste thinks 5 GB per day is generously high.
Chris Cramer says on the surface it may seem that way, but consider the data shows one person served out 250 GB per day. The limit can be adjusted up or down depending on what ITAC recommends.
Molly Tamarkin thinks the penalty should be based on whatever is easiest to administer.
Chris Cramer thinks it would probably be easier to limit a violator for the rest of the semester.
Ken Hirsh reminds ITAC not to forget to encourage rehabilitation among users. He suggests they are punished slightly for the first offense and hope they learn lesson. If not, punish more harshly.
Chris Cramer thinks we should administer this policy/procedure as soon as possible and since the semester has just started, punishment until the end of semester is probably a bit harsh.
David Jarmal asks if we are doing this only to limit bandwidth usage? Does this have anything to do with copyright or intellectual property violations? These are the questions reporters will ask.
Tracy Futhey emphasizes this is only to limit bandwidth. Bandwidth is the issue. Nothing else.
VII. Other business
None