Duke ITAC - September 7, 2017 Minutes

Duke ITAC - September 7, 2017 Minutes

4:00 - 4:10 – Announcements and Introductions

  1. The incoming chair, Ken Rogerson, officially took on the role and warm thanks was given to JoAnne Van Tuyl for her service to ITAC.
  2. Apple has announced that iTunes U will be going away. All content will be moved to Apple Podcasts where it will be available until summer of 2018 and after that, the content will be deleted. The service is no longer viable and usage is low (Duke has around 400 pieces of content, most of it very old). The Office of News and Communication is working to identify next steps for the content. Some content that was created years ago could still be used especially for language courses. Anyone with questions should contact OIT.
  3. OIT Media Technologies provided consultation for the "Portrait of Venice" exhibit at the Nasher Museum of Art which is now open through December 31, 2017.
  4. It was mentioned that capacity planning around network services sometimes uses a metric of one connection per user. However, an observer of a class of students noticed many of the students had smart phones in hand and laptops on desks, both connected to the WiFi network. It's possible they could have tablets or other devices like watches that need internet access and this should be considered in capacity planning.

4:10- 4:20 – Ivy+ Update, Charles Francis, Joe Lopez (5 minute presentation, 5 minute discussion)

What it is:  Representatives from Ivy League schools meet on an annual basis to discuss and share information in various areas. Topics range from overall university directions, budgets, projects, online learning tools, networking and security, and daily operations.

Why it’s relevant:  Sharing experiences and discussing challenges with our peers helps to provide a collaborative environment where ideas are formed and problems are solved. Attendees will share experiences from the most recent conferences.

Duke recently hosted the Ivy+ networking group. These meetings are opportunities to establish benchmarking and help Duke understand how we compare to the other members of Ivy+. The networking area in particular is complicated due to the wide range of approaches schools use for providing networks.

The majority of schools use Cisco products for networking as does Duke. Some of the other schools have focused on the "edge" of the core ("edge" referring to the user "desktop" access point) with intent to increase speeds from 10G to 100G uplink speeds. Most schools had 100G networks configured for Science DMZ (a separate restricted network for high speed transfers such as genome data) but these same schools were not seeing traffic at that level (the highest sustained was reported to be about 10G usage of the 100G network with the more common sustained speed of 1G to 2G). 

In the category of VoIP (voice over IP for telecommunications): Duke is farther along than most of the other schools with 100% telecommunications provided through VoIP. The usage of "softphones" at the universities is very low at 1% ("Softphones" referring to software-based telecommunications applications such as Cisco Jabber that are not restricted to a physical phone and permit users to "answer" calls using a computer). Usage of softphones seems to be concentrated to IT staff. 

A trend among the Ivy+ universities is to out-source the call centers although this is an expensive solution (one university required using two vendors to meet all requirements). Schools are also trying to move away from IPTV ("internet protocol TV") as students are more likely to want content on demand or are more comfortable with "cord cutting" and are familiar with options to view live events (when Duke retired the cable television service, students did not object). The CampusVision service (or "StadiumVision") used by Duke is also of interest to the other universities. One Ivy+ participant did attempt to set up Microsoft Skype for Business but found the investment in backend hardware to be significantly higher than other solutions. Some are also testing getting out of the act of billing as OIT has done (billing is an expensive process in its own right). 

Other key points: 

  • Some schools use a large number of commercial providers or tools.
  • Duke is one of the first to use the next generation of 802.11ac Wave 2 (the fastest current wireless standard).
  • AWS (Amazon Web Service) is also popular as well as the VM Cloud instance. In fact, some are investing in dedicated lines to Amazon facilities and elimination of on-site data centers.  
  • There are efforts to configure the SDN (the Science Data Zone or network) specifically to address network speed bumps based on commodities rather than specific traffic using vendor-based solutions. Duke is measuring the network transfer speeds from building to building (not just to the external network) to see how this could help us as well.

Duke Goals Review and Next Steps

  • The 802.11ac Wave2 wireless deployment is completed
  • Investigate SDN bypass network at the core
  • MCNC network upgrade from 10GB to 20GB for our two links
  • DDoS services through MCNC has been enabled

Questions and Comments:

It may be that phones are not used as must as email or instant messaging options. The mobile phone has also replaced using a "connected" handset. One of the biggest benefits of using a softphone is that it is possible to have a "public" telephone number where calls can be answered on mobile phone with a different number. It is important to note that the physical phones are a network choke point. There are benefits to move to the softphone technology. 

4:20- 4:40 – Raspberry Pi for Wireless MonitoringCharles Francis, Stan Francis (15 minute presentation, 5 minute discussion)

What it is:  The Raspberry Pi is a small form-factor, inexpensive ($35) single board, computer that runs Linux and functions performance-wise much like a standard laptop or desktop. Charles and Stan will explain how Duke is using this new technology to improve wireless monitoring at Duke.

Why it’s relevant:  Diagnosing performance problems with our WiFi environment has been challenging and frustrating for our users.  Raspberry Pi’s acting as proxy “real” users provide us with a low-cost way of keeping tabs on our wireless environment that has already proven useful in improving our service. 

This project came out of the 2016 upgrade of the wireless access points in the dorms that support the many devices using wifi connections including door readers, user computers, mobile devices, etc. OIT did a good job of monitoring infrastructure by checking if a service is up and running. But the problem was how could we monitor the user experience?  Previously, we've relied on support tickets, emails, posts in social media, etc. to determine areas that need attention. This became even more important with the initiative for users to connect to the secure Dukeblue wireless network. We needed a better method to identify bottlenecks and trouble spots and the answer was to simulate the user experience using the “Raspberry Pi”. In the last six months, around 30 of these "Raspberry Pi" computers (computers the size of a deck of cards) were strategically placed in key areas around Duke. Phase I identified the connection experience (locating the access point, connecting to it, authenticating, and transferring data). Phase II looked for unsuccessful or dropped connections to the network. Phase III examined how long it took for the user to connect and how long that connection could be maintained. And metrics continue to be added as the project moves forward and new data points are identified.

The devices were "Raspberry Pi 3" with the following technical specs:

  • CPU: 4× ARM Cortex-A53, 1.2GHz
  • RAM: 1GB LPDDR2 (900 MHz)
  • GPU: Broadcom VideoCore IV
  • Networking: 10/100 Ethernet, 2.4GHz 802.11n wireless
  • Bluetooth: Bluetooth 4.1 Classic, Bluetooth Low Energy
  • Storage: microSD
  • Ports: HDMI, 3.5mm analogue audio-video jack, 4× USB 2.0, Ethernet, Camera Serial Interface (CSI), Display Serial Interface (DSI)

The devices ranged in price from $40 to $60 each depending on configuration and peripherals and ran a full Linux operating system developed for these devices (other Linux distributions were also supported). The Raspberry Pi computers also supported power over ethernet or "PoE" (the ability to be powered from the ethernet connection which permited them to be remotely power-cycled). Only a few of the Pis were configured for “PoE” - the same method used for the security cameras on campus. That said, most of the Pis were hard-wired for power with some testing of battery packs. 

During Phase I, the devices were put in strategic locations where there reports of access issues (either availability or performance). OIT staff developed a method to allow the Pis to behave as "WPA supplicants" which was how these devices connected to the wireless network. The time of the connection was monitored at all stages: connecting, authenticating, authorization, IP number assignment, and completion. The testing indicated that if there was a slow-down, it was not tied to the Radius service (the wireless service provider). Phase II was an expansion to other locations with a focus on a continual connection. The number of devices were increased with improvements in deployment. It was during this phase that we realized these devices could act as a "canary in a coal mine". The Pi devices could serve as an early warning system for wireless issues. 

In fact, the Raspberry Pi computers were able to illustrate a network issue in Giles. The devices were showing connectivity issues while the vendor tools were reporting no problems. The cause turned out to be a recent update to the networking equipment. Once the change was rolled back, the network connectivity issues were resolved for the Raspberry Pi computers. 

The code for the Raspberry Pi computers is stored in a Git repository with a base image installed. Updates are pushed to the devices using Git Continuous Integration. The Pi devices are cataloged in the Configuration Management Database (CMDB). Splunk is housing the logs from the devices and information about them is displayed in customized dashboard using graphs to illustrate link level and signal quality. The Pis also have a faster wireless adaptor capable of running 5ghz that is backwards compatible to 2.4ghz. The devices are connected to the network with both wired and wireless interfaces using static IP routes. There are multiple scripts on the devices that are checking connectivity, recording data, and testing responses for basic network commands such as "ping", http queries, and DNS responses. 

The Splunk dashboards provide graphs indicative of good network function as well as possible problem areas which can be determined geographically because of the metadata from the CMDB. In one situation, the dashboard was also able to illustrate a scenario where faster was not necessarily better in that the device was connected to the faster wireless network but was dropping packets. When the Pi computer switched to the slower network, the connection became more stable.

For the next steps in this project, we will incorporate the Phase I metrics based on the custom development done by OIT in terms of logging connectivity times. We also want to track the number of access points that are detected especially in more commercialized areas that contribute to congestion (which can also be true in certain areas on campus). The devices can also be used to detect "rogue" SSID access points. And speed tests can be conducted using the computers and recorded to Splunk. There is also a possibility of active monitoring using a combination of tools with an end result that an alert can be generated when a Pi device is no longer on the network. 

In addition to the Pi devices, other wireless monitoring approaches include "Via" which is a customer tool that passively records user network performance using javascript for every connection to Duke’s central authentication/login page. Using minimal code and gathering only the IP number and network performance numbers, the data is stored in a centralized collection which shows wired vs wireless connections and fast vs slow transfers. No user information is logged. We are also using a recently acquired tool called "Voyance" which looks at traffic between access points and the controllers and gives us an "inside view looking out" for Duke's network. Voyance is able to show issues with DNS and DHCP immediately and it's very useful in identifying "roaming" issues as well as contention and overlap problems. 

This project was well-received at the Internet 2 conference. The Raspberry Pi computer use case is a unique and economical approach to measuring the user network experience. Since this project started, we've had vendors offer a similar solution at a much higher price point.

Questions and Comments:

Q: For "Via", are you able to get a specific location for where the data originated?

A: We can only get a rough idea by extrapolating the IP number against the custom OIT web utilities. 

4:40- 5:00 – Staff Tech Fair PilotJeannine Sato (10 minute presentation, 10 minute discussion)

What it is:  Jeannine will review findings and lessons learned from the first-ever Staff TechFair in July 2017, a pilot event to engage and educate university employees on end-user IT topics.

Why it’s relevant:  The TechFair served to inform, empower, and support (mostly non-technical) end users on a variety of IT tools and services. The results of this pilot will influence plans to extend this experience to other campus audiences.  We will particularly be looking for ITAC’s guidance on how to best extend this concept to faculty and students.

 

The Tech Fair was a pilot event to test user engagement models with an audience focus on the Duke staff population. The opening of the Technology Engagement Center was a great location and opportunity for directly interacting with our user community. If this was successful, we would then expand to the faculty and student population. We wanted to provide practical topics immediately beneficial to the users without committing the user to large block of time. 

The TechFair was busy with people lined up at the door. We had about a dozen presenters, most associated with OIT with the Law school volunteering to participate as well. We included giveaway items such as coffee mugs for those who attended and individual tables also handed out "goodies". We offered snack foods appropriate for the fair including popcorn, cotton candy, and corn dogs as a way to keep the tone of the event light and fun. 

We had more than 200 staff show up during the two hour event, most during the first hour. There were 32 departments represented. Each table reported that they had roughly 50 meaningful interactions with participants. We also had 48 attendees sign up for the OIT newsletter and we collected 137 netids of attendees who were eligible for the door prize drawings which occurred regularly during the event.  To be included in the drawing, each user had to visit 4 tables and get a card stamped by the presenter. 

During the Tech Fair, the OIT Service Desk staff were available in a side room for more in-depth user engagements. But for the open area, the presenters were challenged with covering a short list of topics, each no longer than 5 minutes that could be covered in a crowded room and still be understandable and practical. 

After the event, the 137 users were surveyed and the vast majority of those who responded indicated they felt this was a good use of their time. 86% said they did learn something new and all respondents indicated they would attend again. Those who commented provided very good feedback - people really seemed to like the event and some wished more of their colleagues had attended. Even those who were familiar with the services OIT offered found new bits of information they could use. Presenters at the event also had positive things to say about the experience and the opportunity to engage directly with users. 

For next steps, the Tech Fair planning team wanted to know if holding this type of event for the faculty would be feasible. The materials would be customized more toward an audience that included faculty, post-docs, teaching assistants, researchers, and other academics. Attendees could walk away from the event in a short amount of time with useful technology tips. Although October was suggested, this month was busy and also was devoted to Security Awareness month. The date of Friday, November 3rd was suggested for this year with the hope that this could be an annual event.

Questions and Comments:

The list of topics looks promising and the organizers are open to any suggestions for other topics that could be covered. There could also be tables that cover broad topics rather than be limited to a particular service.

The material offered to the faculty and research community would need to be modified. One way to do this is work with those conducting the Friday field trips to find out what researchers are doing and what are their needs. There will need to more emphasis on tools for education and research and the professional life while still acknowledging services that help in daily life. There are plans involve CIT who can cover the educational services like Sakai. 

Q: Would it be helpful at an ITAC meeting in the near future to have the faculty bring their devices so that Jabber could be set up and provide input as to what material should be covered in the Faculty-focused Tech Fair?

A: Yes, we could do an informal "test drive" at an ITAC meeting before the actual event.