DevOpsDays Charlotte 2018

Pick up your name badge and swag. Light breakfast and coffee will be available.

I have a two-year-old daughter who adores Dr. Seuss. And as I was reading Cat in the Hat for the 214th time, I realized Dr. Seuss had it all figured out.


His words are odd. The cadence confusing. But there’s a gem hidden in all his children’s rhymes.


You see, Dr. Seuss would have made an excellent engineer.


Because great code isn’t about choosing the perfect method name or building out 95% test coverage. All that is great, but it doesn’t make great code.


YOU DO.


It likely never feels that way. There’s a rhythm to software development that goes something like this: 1. “Easy. I’ve got this.” 2. “Uhhh, maybe not.” 3. “HALP! I have no idea what the f*ck I’m doing.” 4. “How did I not think of that before?!” 5. “I AM A GOD.”


This process is okay if you’re comfortable having a mild psychotic break every sprint. I’m not.


We’re going about it all wrong. Putting ourselves — our egos — above our code. No judgement. I do it too. We’re human. It’s okay.


But I think we can bypass our egos and the emotional ups and downs it produces. This talk will focus on common pitfalls along the development lifecycle and distill Dr. Seuss’s excellent advice into concise steps developers can take before they write a single line of code.


In the words of Dr. Seuss: You have brains in your head. You have feet in your shoes. You can steer yourself any direction you choose. You’re on your own. And you know what you know. And YOU are the guy who’ll decide where to go.

Monitoring systems has traditionally been the responsibility of Ops teams. But if our goal is to align devs, ops, and other roles in the organization, then we need to ensure they are all monitoring critical business systems and do so in a way that takes advantage of the unique perspective that each role offers.
In this session, I’ll break down the expansive monitoring landscape into 5 categories that each provide a unique view of your systems. I’ll show how each category allows your team to have complete observability, avoid blind spots, and work together to quickly resolve performance issues and avoid outages.

As we continue to work on improving our technology processes, there is much we can learn from the discipline of product management. I believe that by applying the techniques and approach of treating your infrastructure and service offerings as products, we can provide a more delightful experience and continuously improve.
Using principles and concepts from people like Marty Kagan and other experts in the product space, I will demonstrate applicable examples of applying the concepts of feedback, planning, and interative improvement to practices including software delivery process, service desk response, and infrastructure as code.

Break

Docker has enabled our teams at Capital One to deploy faster, better, and more modular software, but the technical limitations of Docker also had a surprising effect - it forced teams to break down their monolithic, stateful applications and ultimately improved our organization’s architecture as a whole. Come explore with us as we talk about exactly how this happened and how tools can be a changing force in architecture.

Unfortunately, rolling out changes for humans is not as easy as merging a pull request. How many times have you seen a new project management tool get rolled out, sometimes with much fanfare and polish, and just not get adopted? Have you ever seen your company announce a new business unit which led to a minor revolt? How scary is the word “reorganization”? If you have ideas on how your group’s processes could be better, want to launch a new tool that will work better as your company grows, or have to adjust the way you do things to meet new regulations, learning the basics of change management will help you to get your plans going, launch them effectively, and ensure they stick around.
I’ve helped implement several tools, projects, and process changes over the years. In this talk, I’ll walk you through the basics of organizational change management with specific examples about * Why it’s so hard (Newton’s first law of motion; nobody likes surprises) * Points to consider while implementing (Am I sure I’ve identified all of the stakeholders? What do I do if I can’t satisfy all of their wants?) * Tactics to increase adoption (If you haven’t created evangelists, you’re probably creating enemies) * Keeping the change alive once launched (Can I keep showing improvements? Do I have materials that help newcomers?) * Pitfalls to avoid (Do you really have executive buy-in?)

If you ever need to validate certificates or certificate chains before deploying them, Golang provides a near foolproof test method.
A 3rd party developed a tool that was then handed off to our DevOps team to manage and maintain. Before I could do any re-engineering work, I had to resolve a critical issue—the certificates on the ELBs were about to expire and needed updating.
I assumed that if the ELB, NGINX, or httpd started, it was a good sign. This was a false assumption on my part and I ended up serving a bad chain for a few minutes. This did not break the site, but it was definitely not the way I wanted things to remain.
I needed a tool that would fail if the certificate chain provided was incorrect. I wanted a lightweight tool that could be publicly accessible. Conducting a third-party analysis of the certificates and configuration was a requirement. There were no tools that I could find meeting this need, so I decided to build my own. I turned to the open source language, Golang.
A detailed breakdown of how I built a tiny web server to fit my needs along with what each package is doing as detailed in the article linked above.

Technology organizations have a lot of people that experience deployment or work pain and it comes at the cost of the physical and mental health. What can we do as a community and as leaders to help make sure that DevOps is inclusive of making sure that we talk more about a life work balance as opposed to a work life balance. How do we enable a community with a culture of personal well-being and health so that everyone can provide their best effort?
What can we do from a personal perspective? What can we do from a leadership perspective? What can we do from a coaching perspective?
I will dive into each of these questions to implore our community that focusing on personal health is beneficial to all of us!

Karaoke time!

Habitat is a simple, flexible way to build, deploy, and manage cloud native and modern, distributed applications.  Habitat centers application configuration, management, and behavior around the application itself, not the infrastructure that the application runs on.  It provides automation that can programmatically and declaratively build, deploy, and manage applications and services, both stateful and stateless.  Each Habitat application can be deployed and run on various infrastructure environments including bare metal, VM, containers, and PaaS.


This workshop provides a quick introduction to Habitat during which participants will build, deploy, and run a multi-tier application as a way to explore some of Habitat's application automation features.


Find more about Habitat at [habitat.sh](https://www.habitat.sh/).


Participants should bring a wifi-enabled laptop to the workshop. Participants will be given a remote workstation with all prerequisites installed. The only thing required to access these workstations will be an SSH client (PuTTY on Windows) and familiarity with a interactive text editor (Vi/Vim, Emacs, or Nano).


It is best that participants of this workshop have some familiarity and comfort with the following:


* Writing code (of just about any flavor) in a text editor
* Working on the command line
* Basic system administration – installing packages, configuring those packages, starting service

Databases are special because they’re stateful, and DevOps reflects this: in practice, databases are a lot harder to “DevOps” (as a verb) than other parts of the stack. It’s easy to “DevOps the infra” and “DevOps the web frontend” and so on, because you can just use a declarative tool to make it be the way it should be, and if one of them goes belly up you just kill and replace it. But you can’t do that with the database, and what often happens is orgs get really open-minded about EVERYTHING BUT THE FREAKING DATABASE.
And who suffers? One person often becomes the special database gatekeeper, like Gandalf saying “you shall not pass!” and they suffer. But you know who else suffers? Every developer who now has to coordinate with, get review from, and get permission from the database gatekeeper. Their productivity also suffers.
There’s a better way. It requires cultural support, it requires tooling (like most of DevOps, although it’s not about tools you can’t actually DO it without tools), and it requires a bit of courage. You CAN build tools and processes that help you move faster and be more agile even when – especially when – very large, heavily loaded, mission critical databases are involved. This talk is about how I’ve done it and how my customers have done it.

Drinks, food and good times.

Thought Leaders. DevOps Heroes. Public Speakers. We listen to them as they talk about their solutions, their approaches, and their inevitable triumphs. But are we starting down a dark path, as we forget that 'what makes a great talk' and 'what makes sense for your environment' may not be the same thing? In this entertaining and slightly irreverent talk, the speaker discusses the dangers of taking others' experiences as a source of absolute truth. A discussion of how the innovative and clever solutions that headline various conference talks may very well not apply to your environment will ensue, including but not limited to:

• Matching business requirements to what technologies can deliver
• The trap of feeling like you're falling behind if you're not doing what the "bleeding edge" companies are
• At least one story of how following this approach went hilariously wrong

An introductory presentation on Kubernetes focussed on the relevant interests and concerns of curious DevOps practitioners. The talk starts with an overview of the components that make up Kubernetes and how they work together to form a platform for scheduling and scaling containers. I will then talk through the various resources that Kubernetes make available to the user and how they can be composed together to deploy scalable and robust applications, not only for “cloud native” app but also for existing apps, even the dread legacy apps. I’ll also talk about how it fits in with the existing ecosystem of tools and practices that you’re likely already be using. You’ll leave this talk with a solid understanding of the fundamentals of Kubernetes, how it could fit into your DevOps toolchest, and sense of whether you should use it or not.

Banks are the perfect storm of DevOps anti-patterns where Technical, Organizational and Cultural barriers are all stacked against you.
Cultural sprawl is huge. You don’t have to unlock and influence just one culture; you will encounter a half dozen. As an example, the top 4 banks (Wells Fargo, JP Morgan, Citi, & Bank of America) were 35 separate companies in 1990. They are an average of 175 years old.
In most organizations of almost any size, there are challenges with compliance, tech sprawl, obsolete technologies, and bloated processes. The delivery chain is often split among 3-4 organizations. Culturally, there is fear of change, group overlap and protectionism.
It sounds hopeless, right?
Through long, painful journeys, we learned principles and techniques you can use to weather your storms. These techniques evolved over 10 years in transformations which crossed 3 banks and 4 distinct large-scale transformations. Each transformation was unique, but the principles were universal. They apply to organizations of all sizes, ages and industries.
This talk will detail the principles of how we navigated and influenced cultures of organizations to achieve massively scaled DevOps transformations. We’ll talk about the importance of Leadership and why it is necessary to transcend the cultural challenges in any size organization. Leaders must be willing to go against the grain every day while providing vision, inspiration and protection. We’ll touch on Servant Leadership, Transformational Leadership, Fear of Change, Motivation and Trust.
We’ll explain the “Why” of those principles. From there, we’ll go down a path of applying those principles to actions you can take immediately to begin creating collaborative DevOps cultures.
We’ll discuss how to grow a coalition, anchor behavior, and defend your efforts from those threatened by them. The techniques help incubate your transformation until there is enough critical mass to make it tough to derail.
The three action areas we’ll discuss to help you successfully scale DevOps are: Alliances, Data, & Startup Mentality.
1)Alliances: We’ll discuss how to identify, create and leverage alliances. We’ll show how this helps bypass resistance, plot your strategy and keep your assembly line of on-boards moving.
2)Data: We’ll show how to use data to prove success, communicate weekly, and keep your execs asking for more.
3)Startup Mentality: This keeps your team focused and passionate. I told my teams: “Pretend you are a small company inside a big one. Every day measure our contribution and expect to fold if you don’t deliver.” Acting like a startup means being hungry every day, delivering in increments, focusing on just a couple efforts at a time (80/20), and pivoted rapidly. Deliver small, consistent wins to build a track record and credibility and trust. Focus on customers and their experience.
We struggled, slipped and hit brick walls, but one thing kept us going. Every member of our team believed that what we were doing was the most important thing in the world to our companies. This gave us the courage to keep pushing.
Was it worth it? Yes, every minute of it. We ultimately automated over 800 applications at Bank of America. It can be done in any organization.

Beyond the microservices gordian knot nonsense
Current application theory says that all responsibility for software should be pushed down to the actual DevOps-style team writing, delivering, and running the software. This leaves Enterprise Architect role in the dust, seamingly killing it off. In addition to this being disquieting to EAs out there who have steep mortgage payments and other expensive hobbies, it seems to drop out the original benefits of enterprise architecture, namely oversight of all IT-related activities to make sure things both don’t go wrong (e.g., with spending, poor tech choices, problematic integration, etc.) and that things, rather, go right.
I’ve spoken with several EA teams over the past few years about the changing nature of how EAs can help in a DevOps- and cloud-native-driven culture. I’ll share their experiences including what type of EA help is actually needed and helps organizations thrive.

The military is the ultimate in command and control, making sure your every second is accounted for and every step planned. This is pretty much the opposite of a DevOps culture where sharing and accepting failure are norms… right?
Not so fast…
When I got to boot camp 30 years they made us sit cross legged on bare concrete for “classroom” sessions. Try it for more than 10 minutes some time. It hurts. It made no sense. Of course you get used to it after a few weeks.
Around the sixth week they started teaching us shooting positions. Turns out one of the most stable is sitting cross legged. Stability is important at 300 yards. Something that hurt a couple months earlier was now second nature. We had the foundation required to learn the next steps (literally).
Ready to shoot, right? Yeah, not so much. Now we got to spend a week sitting on grass and pretending to shoot before we ever saw a round of ammunition. Staging deployment anyone?
In much the same way, solid technical principles enable a DevOps culture.
If we make good automation, good test practices, good code quality, useful measurement and helpful feedback loops second nature, we’re free to let our teams solve the hard problems with reduced risk. Command and control can be at the organizational level where it needs to be, with the people best suited to it (our development teams) providing the value required to support that mission.

I’ll introduce “the apocalypse matrix,” which has a severity axis (mild to severe) and a locality axis (personal to global). Preparing for a major global apocalypse (zombies) happens to also prepare you for a mild local apocalypse (snowed into your house with no power for several days). Pivoting this concept to an IT apocalypse will help us come up with plans for incident responses and pack our metaphorical bug out bags.

Sydney Dekker in The Field Guide to Human Error talks about how people often feel that “there needs to be some accountability” when something bad happens, by which they often mean that they should be able to blame the “truly blameworthy”. In contrast to this the local rationality principle talks about the idea that people do what makes sense to them at the time that they are acting, given the goals of what they are trying to accomplish. In these blame seeking moment, motives and reasoning of the people involved are hardly ever taken into consideration and with that the humanity of people is thrown away.
Blame as a whole is anchored in scapegoat theory, hindsight bias and egocentric bias, all of which contribute to the misinformation effect and false memory when trying to actually address why or how something happened. Along with the negative effects that this can have on things like a post incident review, it also can have detrimental effect on the person being blamed as it can cause cognitive dissonance as well as negatively impacting self esteem. This is often an issue between Developers and Operations people. Whether it’s a bug that somehow makes its way to production and takes down the platform or a timeline that slips, there always is someone seeking to blame someone else for what’s going on without taking the time to understand the situation.
With all of these things in mind, this talk will tell the story of blame between Developers and Operations people. It will discuss how stopping at blameless postmortems isn’t enough and willaddress places where blame happens that we may or not be thinking of. In addition it will discuss the related biases (some of which are listed above) that lead us to blame others and how blame negatively impacts individuals, teams, and organization culture as a whole and how that can hinder the DevOps journey. Spring boarding off of that I will offer up some simple things that we all can do to make Developer and Operations relations better along with hopefully making our own worlds better places.

In this talk, I discuss methodologies, pros, cons, and potential pitfalls of the two routes to implementing a new type of technology:

• Formal project process
• Run a skunk works project

This is geared towards the engineer who must realize that technology won’t sell itself. In order to prevent it from becoming abandonware, we need to take certain steps to ensure long-term viability and maintainability.

More organizations are delivering global applications to reach a wider customer base. With those globally distributed applications comes a variety of regulatory compliance and data sovereignty requirements.
In this session, we will explore how DevOps can help your organization automate regulatory compliance and data sovereignty with Kubernetes.

Hailed as The Future by server-smashing conference sponsors worldwide, #serverless might be an option for you, your team, or your company. But serverless – whichever of its definitions you choose – isn’t easy and creates a new set of problems to solve. The AWS Lambda platform, still the most popular option for serverless apps, is one such shiny bag of problems, even as you integrate it with the broader set of AWS services. Documentation provides some advice but its toolbox approach leaves a lot out: which tools should you use for which cases, and how do you make managing it sane? We’ll talk about when you should use Lambda (maybe not as often as you’ve been lead to believe!), how you should organize projects that use it, which tools for it are worth using, and problems to know about as you scale a Lambda infrastructure or attempt to troubleshoot one.
And, I hope, we’ll all leave with the same wistful sense that Amazon’s API Gateway could be so much simpler.