Printed on:
Current at: http://www.turing.org/jon/resume/

Jon Gefaell
(510) 371-4207
Fremont, CA
jon@Turing.ORG

Summary of Qualifications

I am an expert in Unix Networking and Systems Administration with a long term and comprehensive experience in design, implementation and administration of technical operations environments. I have extensive experience with highly available, mission critical, public facing sites. I am also proficient with secure practices, email, DNS, server provisioning, configuration management, monitoring, capacity planning, change control, business continuity and other aspects of technical operations architecture.

Employment History

Senior Unix Systems Administrator
Vindicia, Inc - Redwood City, CA Sept 2015 - current
I joined Vindicia to create architecture for a new datacenter buildout and improve upon Vindicia's mission critical PCI compliant technical operations environment

Improvements to the existing operational environments include:

  • Automated configuration and change management tasks to be performed on hosts across the environment in a safe, reviewable, repeatable and rapid manner. These included timely patching, PCI audits, deployment of corrected and standard configurations, etc.
  • Surveyed host configurations to find a wide variety of hand placed configurations to remove any unique and 'irreplaceable' systems.
  • Created and distributed new standardized configurations and placed them under source control for tracking and documentation through change management.

  • Identified key areas which would be used as the cornerstone of the new data center design and operation. Built an integrated proof of concept ("iPOC") lab to model effective, integrated solutions.

    The 'iPOC' environment includes:

  • Network based host build, IP configuration and configuration management
  • Internal and external DNS with all hosts and addresses accounted for, service aliases, discrete subdomains and standard format zone files, PTR records, multiple third party providers.
  • Management of Hardware/IP addresses and assets information.
  • Comprehensive 'Lights Out' remote operation from host racking through repair, maintenance and replacement.

  • Elements from the iPOC environment made their way into all operational environments:

  • Created internal, managed yum repositories for host builds, upgrades, security patching, third party and internally developed platform software.
  • implemented sophisticated remote TCP syslog with dynamically created directories and logfiles for local and remote application logging of system and local product applications.

  • Most significant achievement

  • Developed build and configuration process for application servers to migrate entire stack from CentOS 6.2 legacy to CentOS 7

  • Linux/Unix Escalations Engineer
    Verizon/Terremark - Cary, NC March 2013 - Nov 2014
    I provided senior level escalation support for Unix systems. This primarily included RTS (return to service activities that could not be resolved by prior support levels. I also handled a variety of escalations for related issues with DNS, network, storage & backup. A focus was placed on identifying and eliminating recurring issues as well as training and mentoring junior level support staff. This included working together to resolve issues as an alternative to escalation. The results were incident reduction, improved MTTR and 'first touch' resolution.

    RTS tasks consumed no more than 50% of work time, often considerably less. Other regular tasks included creating and maintaining standard operating procedure documentation, detailed change instructions and requests, detailed post incident root cause analysis. Provided Unix SME guidance and participation in phone bridges during high impact/severity issues with executive visibility. Worked with account teams (Service Managers, Account Customer Engineers) directly facing customers to resolve problems and suggest and support migrations and upgrades.

    Senior Systems Engineer
    Phoenix 2.0 - San Francisco, CA. May 2012 - March 2013
    On assignment to provide gap coverage for operations engineering at customer site. Built out RHEL Kickstart, improved DNS, reviewed existing product deployment process and configuration management scheme. Provide support and improvements for these processes and other as needed. Also conducted technical interviews for Phoenix consultant candidates.

    Technical Operations Manager
    Adobe Echosign (Formerly Echosign, Inc) - Palo Alto, CA. March 2011 - May 2012
    As always, this was a very hands on position though I took on a larger management role as I built the Technical Operations team and structure from the ground up prior to the acquisition of "EchoSign" by Adobe. The operations team consists of 3 staff engineers and 8 Vendor DBAs. The product technology stack is Java/Apache/AMQ/MySQL running across controlled Preview, Stage & Prod environments in two data centers.

  • Established procedures for safe, predictable and reliable operations such as change control, run books, deployment and theory of operations documents.
  • Established Operations as site deployment owner, achieved zero downtime releases by staging database changes and other careful planning.
  • Developed new Disaster Recovery/BCP plans.
  • Improved data storage, archival and backup methods involving encryption and on/off site storage.
  • Improved network security through comprehensive audit and mitigation.
  • Dramatically reduced site downtime and established 99.99% availability
  • Introduced methods for secure use of sudo, ssh and the ownership and permissions of users and processes.

    Technical Operations Architect, Operations Manager
    Panasonic Cloud Entertainment Company, Unit of Panasonic North America - Cupertino, CA. October 2010 - March 2011
    The company was newly formed as an offshoot of PNA R&D. The premier product was to have been a connected mobile gaming platform known as 'The Jungle'. My task was to create a data center environment for the back end systems managing the device, managing user account, 3g carrier interaction, marketing, app store and so forth. Unfortunately the entire business unit was terminated along with the entire staff when the product was not brought to market.

  • Managed outsourced systems admin team which had built out prototype datacenter and began to introduce necessary changes to move forward to production status
  • Developed Kickstart & RPM Packaging to rapidly provision and update applications allowing ~15 minute automated buildout of core systems including database
  • Organized unified working group between Operations, Development and QA to share methods, tools and skills
  • Successfully demonstrated above with end-to-end OOB device registration and 3g account activation following rapid buildout

  • Infrastructure Architect, Operations Manager
    Philotic/Causes.COM - Berkeley, CA. March 2009 - October 2010
    A very small company with a lot of users (30MM+ Monthly Active users). Causes is one of Facebook's leading applications and helps organizations raise awareness and funds for charitable causes. I took responsibility for all aspects of the company's technical operations. I launched an effort to achieve order and establish best practices in a challenging environment. I was able to drastically reduce outages and measurably improved reliability, availability and serviceability through several strategic and tactical initiatives.

  • Negotiated specifications and contract for new Datacenter & IP Transit
  • Managed equipment refresh including servers, network, power, etc
  • Migrated services to new public IP block, new load balancer configs
  • Built infrastructure including kickstart, puppet, nagios/cacti & snmp
  • Utilized DNS/DHCP/IPAM in datacenter to great effect
  • Achieved extremely agile, certain & safe provisioning and management of servers
  • Built reliable and scalable mail cluster to handle 2Bn deliveries a year

  • Senior Unix Systems Engineer
    Yahoo! Inc. - Sunnyvale, CA. March 2007 - February 2009
    Member of a small team of Systems Administrators responsible for Yahoo's global UNIX and DNS systems Administration for Yahoo! worldwide.

  • Responsible for operating systems provisioning, operational troubleshooting, datacenter tools and services infrastructure.
  • Servers are a mix of FreeBSD and RHEL GNU/Linux on various x86 platforms from small to very large.
  • Specialized in Global Traffic Management and general troubleshooting.

  • Global Infrastructure Operations (Consultant)
    Network Appliance, Inc. - Sunnyvale, CA. March 2006 - March 2007
    Reported to Senior Manager of Global Infrastructure

  • Worked to understand (document) and improve IT infrastructure in areas such as NOC, systems availability and performance monitoring, directory services, email, secure remote access, systems enrollment, OS and Applications provisioning, patching and Maintenance.
  • Review and Recommend new processes and toolsets for comprehensive ITIL integration.
  • Implementation Lead for IBM/Tivoli, Opsware and other key initiatives.
  • Member of Change Advisory Board. Reviewed and approved Enterprise Change Management requests and workflow.

  • Senior UNIX and Network Systems Engineer (Datacenter Architect)
    Netblue, Inc. - Mountain View, CA. May 2004 - January 2006
    Sole systems engineer for rapidly growing internet advertising company

  • Started with existing datacenter operations in 4 ISP Cabinets with ~25 hosts running mostly Redhat 7.2.
  • Enhanced security of operations with wide implementation of ssh/scp.
  • Introduced sudo and other methods to restrict and audit access.
  • Created an automated NOC utilizing redundant servers with Nagios, Perfparse, Cacti, NTOP, Syslog-NG & Logwatch for remote logging & analysis.
  • Host and services provisioned through Kickstart and other automated means.
  • Implemented and Utilized Trac Wiki for Change Management, SVN for source management.
  • Automated backups of infrastructure systems using rsync over ssh.
  • Implemented network and server security with bastion hosts, firewalled network access through host IPTables and Cisco network gear.
  • Provisioned HDS SAN, ISCI equipment & various hardware and software RAID schemes.
  • Programmed Cisco WebNS for load balancing with persistence & redundancy for ~50 websites and other services.
  • Provided on call 24/7 NOC support.
  • Migrated main operations to new datacenter with 6 custom racks in a cage. At the time of my departure there were over 100 CentOS GNU/Linux servers in three locations and three additional Systems Engineers who provide datacenter services which advance key business projects.

  • Senior Consulting Engineer
    Apple Computer - Atlanta, GA. February 2002 - February 2003
    Member of a team providing an overlay of Enterprise UNIX server expertise to Apple's Educational Sales Field Organization.

  • Developed and provided training for Apple's systems engineers and account executives to help them understand and sell Unix servers.
  • Worked with sales teams  to identify and pursue significant opportunities.
  • Developed and delivered presentations and demonstrations to key customers achieving significant success in demonstrating enterprise capability for Apple's UNIX server products.

  • Senior Systems Engineer
    Napster, Inc. - Redwood City, CA. January 2001 - January 2002
    Senior member of Team responsible for design and operations of the server farm.

  • Created reliable, high performance web server farm.
  • Supported large server farm with over 1.7M concurrent users sharing over 280M files
  • Daily operational support of infrastructure (nfs, dns, dhcp, etc...)
  • Linux systems for website(s), napster servers, various infrastructure.

  • Staff Engineer
    Sun Microsystems, Inc. - Mountain View, CA. February 2000 - December 2000
    Converted from contract to regular position. Accepted responsibility as Staff Engineer for all aspects of partner software engineering for Solstice Backup (SBU) product including sustaining product support and new product development.

  • Successfully completed entire major product release lifecycle (front to end) with the release of SBU version 6.0.
  • Participated as member of W-team to coordinate Product development, Customer support, manufacturing and marketing efforts.
  • Authored product Engineering plans, Marketing Requirements Definitions,
  • Responsible for all bugs and engineering escalations.
  • Directed other engineer(s).

  • Contract Positions
    Sun Microsystems, Inc. - Mountain View, CA. February 1996 - February 2000

  • Lead Engineer. Responsible for management of escalated customer support issues. Developed workarounds and product fixes for problems without existing resolutions. Managed Information necessary to coordinate Product development and Customer support efforts.
  • Quality Assurance & System Test Group.Responsible for the testing of Fibre Channel & SCSI hardware RAID devices and software including drivers, GUI management tools, RAID Management software and Veritas Volume Manager/File System. Authored & Developed Test Plans. Automated procedures through jumpstart and rc scripts. Managed group lab.
  • Lead Quality Engineer to perform system software and hardware testing for UltraSparc/AX system board for OEMs. Developed and implemented Test Plans providing 'go-nogo' decisions for product development and marketing.
  • Provided Internet software and hardware expertise to team developing Internet software and HTML/CGI based administration tools. Ported and built various packages to be integrated to products. Managed project lab. Supported Sustaining Engineering for previous product version.

  • Senior Systems Engineer
    Healtheon, Inc. - Palo Alto, CA. July 1996 - December 1996
    Responsible for initial installation and administration of highly secure and available web site, corporate and development environments

  • Installed and configured Gauntlet firewall, Split DNS, NIS with Automount, ISDN and remote access using PortMaster, RADIUS and SecurID token cards.
  • Procured DLT/Legato tape system for database (Informix), UNIX and Windows workstations backups and archival.
  • Worked with Network Engineer to plan and implement network topology for complex multi server facility utilizing multi-homed hosts, Cisco routers and high speed ethernet switches. Renumbered internal networks and migrated from ISDN to T1 Internet connection.

  • Co-Founder & Chief Architect
    Cornerstone Networks - Charlottesville, VA. July 1995 - December 1996
    Co-Founded Internet Service Provider business.

  • Designed and implemented complete system including T1 IP connection, dial-in lines, dedicated client networks, accounting and billing interface.
  • Full services include DNS, Web, FTP and Mail services (with Virtual Domain support) for a mixed consumer and commercial clientele.
  • Equipment included Cisco routers, USR modems, Sparc/SunOS and Pentium/Linux.

  • Business was subsequently sold to a telephone company.

  • Computer Systems Engineer
    University of Virginia - Charlottesville, VA. January 1985 - November 1995
    Responsible for data communications for Administrative Computing Services

  • 1994 - 1995 Designed and implemented community network for county and city government. Project provides Internet access and presence for regional non-profit and government service organizations as well as numerous public use facilities in regional library system.
  • 1993 - 1994 The Hopper project. A public access Internet experiment. (see below; Turing Project)
  • 1993 - 1993 Special Project, Multicast IP Experiments. Created multicast router and application platform for real time audio/video experiments over MBONE meta-network.
  • 1992 - 1994 Netware Lead Support Engineer; lead group of 6 engineers and technicians in the management and support of the campus wide 180 server internetwork. Directly responsible for 24 servers within computer center facilities. Upgraded many systems to 3.x.
  • 1991 - 1992 Designed and implemented statewide network for Continuing Education Division. Over 250 stations at 7 regional centers were connected with Proteon routers and 56Kbps links, and then connected to Internet at UVA. The Centers used SNA, Netware and IP protocols.
  • 1990 - 1991 Installed TCP/IP on IBM SYS/370 to provide FTP and TN3270 services.
  • 1989 - 1990 Designed, implemented and managed 80 station Token Ring network with Netware file and print servers, 3270 gateway, application programs (Wordperfect, Lotus, Dbase, etc?)
  • 1988 - 1991 Continued responsibility for rapidly growing VTAM/SNA network (Now 1250+ nodes).
  • 1987 Began migration to PC based 3270 emulation products.
  • 1986 Responsible for large (750+ node) VTAM/SNA network with local and remote 3270 terminals and printers.
  • 1985 Responsible for processing daily jobs, input and output. Designed and implemented new online system for job submission and review, still in use as of 2002.