Duke ITAC - February 12, 2015 Minutes

I.  Announcements

  • Minutes from the January 15th meeting are approved

II.  Agenda Items

4:05- 4:20 – New Fiber in the Triangle, Elise Kohn (10 minute presentation, 5 minute discussion)

What it is:  Google recently became the third provider to announce plans to offer ultra-fast broadband to residents in the Triangle area, including Gigabit Internet connection speeds. 

Why it’s relevant:  Duke has played a leading role in coordinating the North Carolina Next Generation Network (NCNGN), which has been a primary force behind bringing gigabit broadband to this region. We will talk about how Google’s plans compare with previously announced plans from AT&T and Frontier, how this benefits Duke, and on-going efforts to further innovation using these networks.

  • NCNGN (North Carolina Next Generation Network) is a partnership between Wake Forest University, Duke University, NC State, UNC Chapel Hill, and 6 surrounding municipalities (Carrboro, Cary, Chapel Hill, Durham, Raleigh, Winston-Salem) to bring Gigabit speeds to the Triangle.  
  • A year ago today, Google was in talks with the mayors of these communities to bring fiber to the Triangle.  There has been a tremendous amount of activity, most recently with Google announcing the Triangle area has been selected.
  • Google and AT&T are offering symmetrical 1 gigabit speeds for both uploads and downloads, while Frontier is offering 1Gbps for downloads and 100 Mbps for uploads.
  • AT&T Community Benefits: Up to 100 community centers will receive free gigabit service for 7 years and has agreed to place someone on the ground to engage with the community.  Google has a different engagement model by opting to hire a community engagement manger to determine an engagement strategy. 
  • NCNGN Next Steps – Multi-disciplinary working groups are tasked with developing approaches to advance innovation via efforts like:
    • Collaborating on specific research projects/grant proposals
    • Creating competitive grant programs
    • Unique fellowship & internship opportunities
    • Hackathons
    • There are unique opportunities to translate research into real-life demonstration projects typically available to researchers on campus.  If you have ideas about working with NCNGN, contact elise.kohn@duke.edu.

Questions and Comments

Question: Do we suspect that Frontier is releasing a competitive service offering for Google and AT&T?  Frontier may not be prepared with their infrastructure and awareness to offer a similar service offering. 

Question: Will this be a spotty rollout in neighborhoods? Google’s model is to work with city leaders to define fiber-hoods and rally residents to get enough interest to deliver service.  AT&T will be spottier and will differ in Durham than in other communities especially since they’re new to the area.  However, outside of Durham, it will depend on the status of their existing infrastructure. 

Question: Is AT&T’s low speed service also going to be free? No.  There are certain areas where they will provide free low speed service to low income housing units/apartment buildings, but they will not follow the Google model.  However, Google is only presenting two speed options whereas AT&T has a more tiered approach to offering service and may offer a wider variety of price points. 

Question: Is there any chance this could be derailed by political vested interest?  Since this partnership is through private industry, no carrier has been given preferential treatment, hence any other carrier has the liberty to provide similar service offerings.  


4:20- 4:45 – CSG Update, Tracy Futhey, John Board, Charley Kneifel, Mark McCahill (15 minute presentation, 10 minute discussion)

What it is:  The Common Solutions Group works by inviting a small set of research universities to participate regularly in meetings and project work. These universities are the CSG members; they are characterized by strategic technical vision, strong leadership, and the ability and willingness to adopt common solutions on their campuses.

Why it’s relevant:  CSG meetings comprise leading technical and senior administrative staff from its members, and they are organized to encourage detailed, interactive discussions of strategic technical and policy issues affecting research-university IT across time. We would like to share our experiences from the recent 2015 Winter meetings.

  •  Software Defined Networking Session: We are ahead of our peers so much so that the state of our SDN offering was not even on the survey. 
    • Switchboard allows authorized users to make a fast bypass connection between two points  (recording)
    • Another theme in SDN was there wasn’t an anticipated use of the hybrid mode switches.   The importance of instrumentation of the network
    • On-demand configuration of the network connecting to AL2s.
    • Openflow Managent Gatway (OMG) it has a second SDN controller running in parallel with whatever software is currently running.
    • Researchers would prefer to slice the AL2s network on demand.  Duke has 2 10GB connections to AL2s.
    • Devox – Automation in support of quickly deploying – In the past updates to Stanford’s websites was cumbersome.  They have changed that so that the content owners check in their content into a repository, their environment recognizes the change and once approved, automatically loads the new page.  Duke will be
    • 10 years ago, outages were taken for granted that they would exist.  Now, those are unacceptable and there is an expectation of zero downtime.  As an example, Rob Carter is deploying a Kerberos upgrade and has built… across multiple servers to determine when something changes.  Berkley has a framework in place for administrating desktops with significant amounts of app integration testing on a regular basis.
    • Enterprise Service – Workday was driving many organizations (cloud-based HR Finance system), but relies on batch file updates for data.  UPenn has an organization for developing student apps and all previous apps are maintained.
    • Workshops: Data driven decision making and business intelligence, research computing directions


Questions and Comments

Question: Are there any implications of network neutrality? ISPs and networks are determined to be a telecommunications function.  What implications will that have….

Comment: Who might have need for higher levels of bandwidth and lower levels of latency?  We would be eager to know about people and research cases so we can apply point solutions. 

Question: Is this approach sustainable? UPenn was one of the early Hackathon implementations.                     

Question: What is the process for decommissioning apps?  Is that a collaborative efforts between the students and central IT? The student


4:45- 5:05 – Strategic Planning Update, Julian Lombardi (10 minute presentation, 10 minute discussion)

What it is:  We will provide an update on the strategic planning efforts and discuss how ITAC will contribute to these efforts.

Why it’s relevant: As an early phase of university-wide strategic planning, many sectors of the university including IT are being asked to identify major issues likely to inform the planning effort; IT leaders are engaging in a SWOT analysis of services and systems at Duke.

  •  A SWOT Analysis is a framework to help identify strategic issues for strategic planning.  Specific statements are gathered from the members of the committee to include Strengths, Weaknesses, Opportunities and Threats.
  • Strengths
    • The existence of ITAC is a strength.  The conversations that happen here.  Issues are raised and addressed regularly.
    • People who are less connected with IT than we are have an impression that when something is broken, it doesn’t get fixed.  There is a disconnect with a large faction of users, including faculty and the IT organizations.  Users think our processes are too hard and don’t realize how centralized support is here at Duke.  Effective collaboration and cooperation among IT professionals in a decentralized environment, but the disconnect is with the customers.    Most people don’t know which group of IT service staff to go to.  There is a willingness to help, but Service Now interfaces are difficult to use.  Is it a request or is it an incident?
    • Failure to make services transparent. – easy to access, easy to use and well publicized. 
    • There is a tendency of faculty to be cynical and dismissive of big systems and central organizations.  But there are many real life experiences that feed that sentiment 
    • Weaknesses
    • What is meant by poor support for interdisciplinary support?  Those groups/units tend to be left out of IT support and there is no budget
    • Absence of a faculty associated strategic planning committee for the future of online educations – e-learning roadmap group no longer exists and no vision. 
    • Training of local support staff:  We have an attitude of customer service, but could there be training on not just customer service, but on innovative solutions for solving problems for proactive support based on better training; keeping them aware of new trends and new needs and possibly punt something off to a central organization where necessary; knowing how/when to triage/refer. 
    • There isn’t very much input in the hiring process for local IT network support staff.  There may or not be enough knowledge of the complexity of modern IT needs for decision makers on campus.  Variability of hiring at the local level and.  It’s not just on the local level.  For instance A/V is a big problem. The temporary solutions are not very effective because.  There is a lot of inertia in the problem.  A projector when bad in a conference room – which department pays for it and pays for its replacement.  By the retreat of certain groups, a hole is left; lack of clarity and responsibility.  An outcome of this analysis could be to change the model and rely less on local support and more on central support with
    • Opportunities
      • What does the last one have to do with IT?  Anything we do globally has an IT component
      • The SOM has now begun to actively force every PI to come up with a data management plan.  Almost everyone in the room’s immediate response was how can we plan our data management if we have to place to put it?  This is an opportunity to create something centrally to archive and manage our data.  The proliferation of open source and access has to
      • Media repository and curation of media assets.  We do not have a means for people to share them
      • Transparency in terms of greater data access and availability. To provide university data to researcher to do interesting things with.
      • Threats
        • Global expansion because it continues to stretch our resources and our ability to support those efforts and maintain support for our local campus.

Questions and Comments

5:05- 5:30 – Denial of Service Attacks against www.duke.edu, Richard Biever, Charley Kneifel (15 minute presentation, 10 minute discussion)

What it is:  The recent denial of service attack against www.duke.edu had a much larger impact in that it also affected central campus services such as DNS and DHCP.  

Why it’s relevant:  We will discuss the timeline of the attack, why it affected other campus services, and how we mitigated the impact by moving www.duke.edu to Amazon Web Services.

  •  Attacks occurred between January 22-23
    • January 22 (morning) and Syn flood attack occurred
    • January 22(afternoon) – NTP reflection attack
    • January 23 (morning) – Syn flood attack

Syn flood attack starts by sending a SYN packet with a spoofed return IP address.  The attack went through the Data Center VRF (DNS and DHCP underlying network services).  The spoofed IP computer sends an ICMP reply back

The firewall was taken out of commission (web server, DNS, DHCP were fine) but the firewall blocked it

Corrective action

January 22 – Increased connection limits on firewall, blocked NTP to the Duke and Chapel Websites, blocked external DNS queries to the internal caching name servers, and prepped moving www.duke.edu to AWS

January 23 – the web server was moved to AWS.

The attackers picked victim computers across the world. 

Elastic compute cloud and as of yesterday about 75% of the traffic went to Amazon.  All of the infrastructure we’re deploying with Docker images and containers make it easy for us to migrate these services to Amazon.

Questions and Comments

Question:  Are there downsides to leaving it with Amazon?  This month we spent $38 for Amazon to host it.  We’ve had to change some of our monitoring processes, but we’re not seeing a large downside.  

Comment: If Amazon has a bad day, we will too.  But by packaging it up in containers, it allows us to move the services between providers.

Question: A Denial of Service attack could happen at any web server? Not just any web server, but any server.  This is common response to making public statements that offend groups.   Lots of malformed packets…There are things we can do to separate DNS traffic and replicate services.