Site Reliability Engineering

The Problem

The traditional approach consists of having a team of sysadmins who are responsible for deploying the services in production and doing the maintenance.

As illustrated in the following diagram, 2 separate teams, the software team developing the applications, and the sysadmin teams, deploying and maintaining the application in production.

The problem with this approach, is that as more services are integrated or developed, the sysadmins team needs to grow exponentially, incurring an increase on direct costs.

And on the other hand, there is indirect cost at the organization level, most of the time, software teams and sysadmins teams have different objectives, priorities and use different terms to define the same event.

Google over the years, provided more and more services, and experienced directly the limitations of the traditional approach.

With the implementation of SRE Teams (Site Reliability Engineering), they were able to :

(1) : increase substantially the velocity of features/services deployed in production

(2) : improve the service reliability in production

SRE Principles

4 main principles govern the SRE model and describe how organization can implement them in their processes:

Embracing Risk

Service Level Objectives

Eliminating Toil

Monitoring Distributed Systems

Embracing Risks

Ideally your service is available all the time for your users. But unfortunately, sometimes you can experience unplanned downtime (failures, etc) or you may need to perform some updates (planned downtime).

So you can't really have a 100% service availability. What you can expect is having a number closer, generally the 5 nines: 99.9%, 99.99%, 99.999%.

Experiences have shown that trying the reach absolutely the highest service availability cost you in terms of agility (less frequent updates, few features, etc..)

As illustrates in the following diagram the best way to improve your service overall cost is by balancing the risks, as illustrated in the following diagram :

The first step will be to measure the service risk, trying the get the uptime level. The best formula to calculate the availability of the service :

availability = successful requests / total requests

In a second step you can prioritize the impact of the services on your business (revenue). A service to free users, will for instance have less impact than a service offered to premium users.

At a third step, you can define the target level of availability. As illustrated in the following diagram, to clearly define the service risks, product owners and software teams can agree on the impacts if the service is unavailable.

Below is a table including the availability level and corresponding unavailability window per year and per day to give you a good understanding.

Let's understand it better with real use cases.

Use Case 1 : A service that serves 5 millions requests/day and has a target availability level of 99.9% per day, can tolerate up to 25000 errors (or failed requests) per day and still meets the service level target.

Use Case 2 : A service that serves 2 millions requests/day and has a target availability level of 99.99% per day, can tolerate up to 2000 errors (or failed requests) per day and still meets the service level target

Service Level Objectives

Additionally to the SLA (Service Level Agreement) which is an indicator more appropriate for the business requirements, SRE introduces 2 types of metrics to measure the service:

SLI (Service Level Indicator), which is a specific metric to measure the service

There are 2 categories of SLI metrics : User facing system and Storage.

---

User facing system

Latency

the time it takes for the service to respond to a HTTP request (1ms, 100ms, 500ms)

Availability

the number of failed requests in proportion to all the requests (2% failed HTTP request per/s, 10%, ..)

Throughput

the number of requests or queries per/s (QPS) the system can handle (1 million requests/s, 5 millions requests/s)

---

Storage system

Availability

define whether the data is accessible.

Latency

the time it takes to read/write data (in terms of IOPS)

Durability

the ability to store information with no disruption, errors or failures.

How to define the SLO ?

You can follow these 3 steps to define your service level objectives.

Eliminating Toil

While running application in production, you may have to perform some operations: manual, repetitive, or automate; like running a script to automate some tasks, etc ;

The Toil represents basically the operations (cited above) that tend to increase exponentially as your service in production scales as illustrated in the following diagram.

Google approach is to allocate 50% of the time of SRE operations spent on reducing the toil, as illustrated in the following diagram :

Time allocated to reduce toil, is like an investment on the future of your platform operations; Not allocating enough time on toil, can significantly increase the time you spend only on toil, reducing the time you allocate for other tasks (increasing delay for fixing issues, or developing features, etc..)

Monitoring Distributing Systems

Monitoring systems today get more and more complex. The old ways to conduct monitoring to get alerted if something breaks, do not apply to large distribution systems.

The evolution of software architecture (monolithic to microservices), has brought new challenges and to monitor efficiently these large scale distributed systems, there are few rules to adopt.

There are 4 crucial metrics to help you track how healthy is your system :

---

Latency

How long your system takes to respond to a request ? It's usually expressed in terms of milliseconds;

---

Traffic

How many requests your system is handling? For instance for a web application it might be how many HTTP requests per second ?

---

Errors

How many incoming requests fail ? How many HTTP errors (5xx, etc,..). This indicator can help you determine how reliable is your system.

---

Saturation

This indicator will determine if your system capacity is overloaded (max CPU, memory, disk consumption).

---

Some rules are important to keep in mind to maximize the effectiveness of an incident response, but also to prevent SRE teams to stay constantly under pressure.

Automate incident response as much as possible : You may receive several alerts for incidents that require the same manual resolution. Make a list of these incidents and make sure whether or not they are eligible for an automation (script or any other automated task) to prevent triggering unnecessarily your alerting system

Keep things simple : There is no reason to track a metric on everything. Some minor incidents not affecting the user experience or that may have lower impact on the service availability, might be removed from the alerting system. Your team receive several alerts, if there are too many of them in a short period, this might create fatigue.

Dashboards for what really matters : Too many indicators on your dashboard might be inefficient. Your dashboard should remain as simple and clear as possible to catch easily an incident, a pattern, a problem.