Site Reliability Engineer (SRE)

SRE is what happens when you ask a software engineer to design an operations function. It is a discipline that incorporates aspects of software engineering and applies them to infrastructure and operations problems, with the primary goal of creating scalable and highly reliable software systems.

DevOps is a cultural philosophy focused on breaking down silos between development and operations. SRE is a specific implementation of DevOps. To use a common industry analogy: 'class SRE implements interface DevOps'. SRE provides concrete metrics and practices like error budgets and SLIs to achieve DevOps goals.

Traditional ops often relies on manual intervention (sysadmin work) and sees change as a threat to stability. SRE uses automation to replace manual work (toil), embraces 'risk' through error budgets, and treats infrastructure as code, allowing for faster, safer releases.

Key responsibilities include Availability, Latency, Performance, Efficiency, Change Management, Monitoring, Emergency Response, and Capacity Planning. SREs divide their time between operational 'on-call' tasks and engineering project work to improve system reliability.

The error budget is the maximum amount of time a technical system can fail without contractual or user-facing consequences. If your SLO is 99.9%, your error budget is 0.1% (about 43 minutes of downtime per month). It is the bridge between product velocity and reliability.

Error Budget = 1 - Service Level Objective (SLO). For example, if you have 1 million requests per month and an SLO of 99.9% success rate, you are 'allowed' 1,000 failed requests before the budget is exhausted.

When the budget is spent, the team typically halts all new feature releases and focus entirely on reliability improvements and bug fixes until the service is back within the SLO. This enforces a 'reliability first' culture.

An SLI is a quantitative measure of some aspect of the level of service that is provided. Examples include request latency, the number of errors per second, or system throughput.

An SLO is a target value or range of values for a service level that is measured by an SLI. For example: '99.9% of HTTP requests should return with a 200 OK status code.'

An SLA is a legal contract between a service provider and an end-user that defines the expected level of service and the consequences (usually financial) if those levels aren't met.

SLI is what you measure (latency), SLO is the target you want to hit (99.9% < 200ms), and SLA is the contract that says what happens if you miss (refunds). SREs focus on SLIs and SLOs to avoid hitting the SLA.

Common SLIs include Availability (Success rate), Latency (Response time), Throughput (Requests per second), and Error Rate. For storage systems, durability and data consistency are also critical SLIs.

Setting SLOs requires collaboration between SREs, developers, and product managers. You look at historical performance, user expectations, and business requirements to find a target that is challenging but achievable and meaningful to the user experience.

The target depends on the criticality of the service. 99.9% (three nines) is standard for many web apps, while 99.99% (four nines) is for core infrastructure. Each 'nine' added increases the cost and engineering complexity exponentially.

Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.

Toil is measured by tracking time spent on manual tasks like ticket handling, manual restarts, and data migrations. SRE teams often use time-tracking or survey-based 'Toil Audits' to identify candidates for automation.

Google's SRE model recommends a maximum of 50% toil. The other 50% should be spent on engineering projects that reduce future toil or improve the system's scalability and reliability.

Toil is reduced through automation (CI/CD pipelines, self-healing scripts), improving documentation to enable self-service for developers, and simplifying system architecture to eliminate manual intervention points.

It is the rule that at least 50% of an SRE's time must be dedicated to project work that adds permanent value to the system. This prevents the team from becoming a traditional operations 'support' center.

A blameless postmortem is a formal review of an incident that focuses on identifying systemic failures and process gaps rather than pointing fingers at individuals. The goal is to learn and prevent recurrence, not to punish.

It should include: Executive Summary, Incident Timeline, Root Cause Analysis, Impact (User/Business), What went well/badly, and most importantly, Action Items with assigned owners and deadlines.

An incident is any unplanned interruption or reduction in quality of service. An outage is a severe incident where the service is completely unavailable to all or a large subset of users.

The average time it takes for the monitoring systems or staff to become aware that an incident is occurring. Improving observability helps lower MTTD.

The average time it takes to fix a system after a failure. This includes the time spent diagnosing the problem and the time spent implementing the fix.

A measure of a system's reliability, representing the average time the system operates without any incidents. Higher MTBF indicates a more stable system.

Monitoring tells you that something is wrong (e.g., CPU is high). Observability allows you to understand why it is wrong by looking at the internal state of the system through logs, metrics, and traces.

1. Metrics (Aggregated data over time). 2. Logs (Immutable records of discrete events). 3. Traces (End-to-end journey of a request through microservices).

Metrics are cheap to store and good for alerting. Logs provide deep context but are expensive to store. Traces help identify bottlenecks in distributed systems across service boundaries.

Defined by Google SRE, they are: Latency (time to service request), Traffic (demand), Errors (rate of failed requests), and Saturation (how 'full' your service is).

Latency is the time it takes to service a request. It is important to distinguish between the latency of successful requests vs. failed requests (errors can sometimes be very fast or very slow).

Traffic is a measure of how much demand is being placed on the system, measured in high-level service-specific metrics like HTTP requests per second or concurrent sessions.

Errors measure the rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (e.g., HTTP 200 with wrong data), or by policy (e.g., requests that take > 1s).

Saturation measures how full your service is. It highlights the most constrained resources (e.g., CPU, Memory, or Database pool size). Most systems degrade in performance before reaching 100% saturation.

I would monitor HTTP request rates, status code distributions (2xx vs 5xx), P95/P99 latency, active user sessions, and resource utilization (CPU/RAM) of the app servers.

Key database metrics include Query Latency, Connection Pool saturation, Lock contention, Disk I/O wait times, and Replication Lag (if using read replicas).

Node CPU/Memory pressure, Pod restart counts, API server latency, Kubelet health, and Persistent Volume usage levels.

White-box monitoring is based on internal telemetry (logs/metrics from the app). Black-box monitoring tests the system from the outside (like a user) to check for visible behavior and availability.

Prometheus is an open-source monitoring tool that uses a 'Pull' model to scrape metrics from targets at regular intervals and stores them in a time-series database.

Prometheus stores data as time series: streams of timestamped values belonging to the same metric name and the same set of labeled dimensions (key-value pairs).

PromQL is the Prometheus Query Language. It allows you to select and aggregate time series data in real-time to create alerts or visualize performance in Grafana.

`rate(http_requests_total[5m])`. This calculates the per-second average rate of HTTP requests over the last 5 minutes.

`sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))`. This gives the ratio of 5xx errors to total requests.

`histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))`. This calculates the 95th percentile latency from a histogram.

Grafana is an open-source visualization and analytics platform. It connects to Prometheus as a data source to create dashboards that track SLIs and SLOs in real-time.

The Pushgateway is used for short-lived batch jobs that cannot be scraped (pulled) by Prometheus because they finish before the scrape interval occurs.

Service discovery allows Prometheus to automatically find and scrape new targets (like pods in Kubernetes) as they are created, eliminating the need for manual configuration.

It is a popular logging suite. ELK is Elasticsearch (storage), Logstash (processing), and Kibana (viz). EFK replaces Logstash with Fluentd/Fluent Bit for a lighter Kubernetes footprint.

Structured logs are in a machine-readable format like JSON, making them easy to query and parse. Unstructured logs are plain text strings, which are human-readable but hard for machines to analyze at scale.

A method used to profile and monitor applications, especially those built using microservices architecture. It tracks a single request as it passes through multiple services.

These are open-source distributed tracing systems used for monitoring and troubleshooting microservices-based distributed systems.

A Span represents a single unit of work (e.g., a DB call). A Trace is a collection of spans that represent the entire journey of a request from start to finish.

Tracing every single request is too expensive. Sampling is the process of selecting only a percentage of requests (e.g., 1%) to record traces for, providing enough data for analysis without overwhelming the system.

By including a unique 'Trace ID' in the metadata of logs and metrics. This allows an analyst to jump from a high-latency metric to a specific trace, and then to the exact log lines for that request.

Cardinality refers to the number of unique combinations of label values for a metric. High cardinality (e.g., using 'User_ID' as a label) can crash your monitoring system by consuming too much memory/storage.

Alert fatigue occurs when people are overwhelmed by too many non-actionable alerts. It is prevented by only paging for SLO-violating incidents and ensuring every alert has a clear runbook and resolution path.

A good alert must be actionable, symptom-based, and relevant. It should clearly indicate that a service is failing or about to fail (SLO violation) and provide enough context for the responder to take immediate action. If an alert doesn't require a human to do something right now, it should be a notification, not a page.

An Alert (or Page) is an urgent interrupt that requires immediate human intervention, usually sent via PagerDuty. A Notification is non-urgent information (like a successful backup) usually sent to Slack or email for later review.

Common levels are: Critical/P1 (Immediate page, service down), Warning/P2 (Symptom-based, investigate during business hours), and Info/P3 (Informational, no action required).

By following a template: 1. Summary of what is broken. 2. Impact on users. 3. Link to the relevant Grafana dashboard. 4. Link to the specific Runbook for resolution.

The process of sending an alert to the specific team or individual responsible for that service. For example, database alerts go to the DBA/Data-SRE team, while frontend alerts go to the UI team.

A set of rules that defines who to contact if the primary on-call person does not acknowledge the alert within a certain timeframe (e.g., if P1 doesn't answer in 15 mins, page the secondary).

These are incident management platforms that handle on-call schedules, alert routing, and automated escalations via phone calls, SMS, and mobile apps.

1. Delete low-signal/noisy alerts. 2. Turn pages into Slack notifications if they aren't urgent. 3. Fine-tune thresholds. 4. Automate the resolution of recurring issues using scripts.

Suppression is pausing an alert during known maintenance. Inhibition is a Prometheus feature that stops a 'Low' alert from firing if a 'High' alert for the same target is already active (e.g., don't alert 'Disk Slow' if 'Server Down' is firing).

Flapping occurs when a metric rapidly crosses the alert threshold back and forth, causing a flood of 'Firing' and 'Resolved' notifications. This is usually fixed by adding 'Hysteresis' (a buffer) or a duration requirement (e.g., 'for: 5m').

By using trend-based alerting (rate of change) instead of static thresholds, and ensuring alerts are based on user-facing symptoms (SLIs) rather than internal causes like high CPU.

A schedule where team members take turns being responsible for production incidents. Common patterns include 24/7 weekly shifts or 'follow-the-sun' where teams in different time zones handle daytime hours.

Ideally, a rotation should have at least 8 people to ensure engineers aren't on-call more than 25% of their time. This prevents burnout and allows for sufficient 'recovery' time.

By enforcing a 'no project work' rule during heavy on-call shifts, ensuring the next day off after a rough night, and constantly automating away the issues that cause the pages.

1. Acknowledge the alert. 2. Verify the impact (is it real?). 3. Check the Runbook. 4. Communicate status to stakeholders. 5. Mitigate the issue (e.g., restart, rollback). 6. If stuck, escalate to secondary or a specialist.

Using tools like Ansible or Lambda to automatically execute the steps in a runbook when an alert fires, such as clearing a temp directory when a disk space alert reaches 90%.

Description of the alert, diagnostic commands (what to check), common fix steps, dependencies (what this service relies on), and escalation contacts.

Page-worthy: Users can't pay, site is 404, checkout is broken. Non-page-worthy: One of ten nodes is slightly slow, a background cron job failed (but will retry).

By 'Blast Radius' (number of affected users) and 'Criticality' (is it a core feature like login vs. a cosmetic one). SREs focus on the P1/highest-impact incident first.

Grouping multiple related alerts into a single incident to reduce noise. For example, if a switch fails, don't page for every individual server behind it; page for the switch failure.

A scheduled time for updates. Alerts should be 'silenced' or 'suppressed' during this time to prevent the on-call engineer from getting unnecessary pages for expected downtime.

By tracking 'Alert Density' (alerts per shift), 'Actionability' (how many alerts resulted in an actual fix), and 'MTTA/MTTR' metrics.

The time from when an alert fires to when the on-call engineer marks it as 'Acknowledged,' indicating they are starting to work on it.

The total time from the start of the incident until the service is restored to its normal, healthy state.

Through better observability (faster debugging), comprehensive runbooks, automated rollbacks, and regular 'Game Day' exercises to practice response steps.

An incident is an unplanned event that disrupts or reduces the quality of a service. In SRE, we primarily care about incidents that threaten our SLOs.

SEV-1: Critical (Site down). SEV-2: High (Major feature broken). SEV-3: Medium (Partial degradation, workarounds exist). SEV-4: Low (Minor bug/cosmetic).

SEV-1 is an 'all hands on deck' emergency. SEV-2 requires immediate attention during work hours. SEV-3 is handled via the standard ticket backlog.

Detect -> Acknowledge -> Triage (Severity) -> Mitigate -> Resolve -> Postmortem.

Incident Commander (IC): Leads the response. Scribe: Documents the timeline and decisions. Comms: Updates stakeholders and customers.

The person with ultimate authority during an incident. Their job is not to fix the bug, but to coordinate the experts, clear roadblocks, and ensure a path to mitigation.

Assigning tasks to responders, deciding when to escalate, managing the 'War Room' communication, and declaring when the incident is mitigated.

The plan for internal (leadership) and external (customer) updates. Updates should be frequent (e.g., every 30 mins for SEV-1) and transparent about what is known and what is being done.

Use a dedicated Slack channel for responders, a 'bridge' call for high-severity syncs, and a public Status Page for end-users.

A public website that shows the current health of services. It builds trust with users by providing official updates during outages.

A chronological record of events, including when the issue started, when it was detected, what actions were taken, and when it was fixed. This is essential for the postmortem.

By keeping the call focused. The IC leads, responders speak only when they have data or have finished a task, and side-discussions are moved to Slack to keep the 'airwaves' clear.

Bringing in more senior engineers, other teams, or executive leadership when an incident is too complex or has extreme business impact.

1. If the current team cannot identify the cause. 2. If the mitigation steps failed. 3. If the incident has lasted longer than a predefined threshold (e.g., 1 hour).

Mitigation is 'stopping the bleeding' (e.g., failing over to a backup). Resolution is fixing the root cause so the issue doesn't happen again.

Rollback: Reverting to the previous stable version (safest). Roll-forward: Applying a quick 'hotfix' patch to fix the bug in the new version (riskier).

A meeting and document created after an incident to analyze what happened, why it happened, and how to prevent it in the future.

A culture where the focus is on improving the system rather than punishing the person who 'typed the wrong command'. It encourages honest reporting of mistakes.

An iterative process of asking 'Why?' five times to move past symptoms and find the actual root cause of a failure.

The technical deep-dive to find the fundamental reason for an incident (e.g., finding the specific code bug or hardware failure).

Root Cause is the main trigger. Contributing Factors are things that made the situation worse, like a slow monitoring system or a missing runbook.

Context, user impact, timeline, RCA, what went well, what went wrong, and corrective action items.

Concrete tasks aimed at preventing recurrence. They should follow the SMART criteria (Specific, Measurable, Achievable, Relevant, Time-bound).

Using a bug tracker (Jira/GitHub Issues). SREs often ensure that 'Postmortem Actions' are prioritized in the next sprint to ensure the engineering work actually happens.

By completing postmortem action items, implementing chaos engineering, and improving automated testing in the CI/CD pipeline.

A broader team meeting to discuss the findings of a postmortem, share knowledge, and ensure all stakeholders agree on the remediation plan.

By treating every outage as 'free training'. We document the lessons and share them via an 'Incident Library' or internal newsletters.

Chaos engineering is 'intentional' incidents. We break things on purpose in production (like killing a node) to ensure our systems are resilient and our response process works.

A scheduled event where a team practices responding to a simulated failure (e.g., 'What do we do if the primary database region disappears?').

A large-scale test of the organization's ability to recover from a catastrophic failure, usually involving the actual failover of production traffic to a secondary region.

Highly available (HA) systems are designed by eliminating Single Points of Failure (SPOF). This involves using multi-AZ/region deployments, load balancing, redundancy at every layer (web, app, DB), and implementing automated failover mechanisms.

High Availability focuses on minimizing downtime and ensuring the service is 'up' as much as possible. Fault Tolerance goes further, ensuring that the system continues to operate correctly even when components fail, often with zero perceptible impact on the user.

Redundancy is the duplication of critical components or functions of a system with the intention of increasing reliability, usually in the form of a backup or fail-safe.

Active-Active: All nodes handle traffic simultaneously, providing better performance and seamless failover. Active-Passive: One node handles traffic while the other is on standby; the passive node only takes over if the active one fails.

The process of distributing incoming network traffic across multiple servers to ensure no single server is overwhelmed, which improves both responsiveness and availability.

Common algorithms include Round Robin (equal distribution), Least Connections (sends traffic to the least busy server), IP Hash (sticky sessions based on IP), and Weighted Round Robin (accounts for server capacity).

A mechanism where the load balancer periodically pings or requests a specific endpoint (e.g., `/health`) on backend servers. If a server doesn't respond or returns an error, the LB stops sending it traffic.

A design pattern used to detect failures and encapsulate the logic of preventing a failure from constantly recurring during maintenance or temporary outages. It 'trips' (stops calling the service) to allow the failing service to recover.

Retry logic attempts a failed operation again. Exponential backoff increases the wait time between retries (e.g., 1s, 2s, 4s, 8s) to avoid overwhelming a struggling service (thundering herd problem).

Restricting the number of requests a user or client can make to a service in a given timeframe. This protects the service from abuse and ensures fair resource distribution.

Rate Limiting is the policy (e.g., 100 req/min). Throttling is the action of slowing down or rejecting requests once that limit is reached.

Determining how and where to store temporary data (e.g., Redis, CDN) to reduce latency and database load. Common strategies include Cache-aside, Read-through, and Write-through.

The process of removing or updating cached data when the source data changes. This is notoriously difficult ('There are only two hard things in Computer Science: cache invalidation and naming things').

A Content Delivery Network is a distributed group of servers that caches content near end-users. Use it for static assets (images, JS, CSS) or high-traffic global websites to reduce latency.

Copying data from one database server (Leader/Master) to one or more other servers (Followers/Slaves) to improve read performance and provide high availability.

Master-Slave: One node handles writes, others handle reads. Multi-Master: Any node can handle writes. Multi-master is much harder to implement due to conflict resolution requirements.

A horizontal scaling technique where data is split across multiple independent database instances based on a shard key (e.g., User ID), allowing for massive datasets and throughput.

Vertical (Scale Up): Adding more CPU/RAM to a single server. Horizontal (Scale Out): Adding more servers to the pool. SREs prefer horizontal scaling for its better reliability and cost-efficiency.

Stateless: The app doesn't store client data locally; any node can handle any request (easy to scale). Stateful: The app remembers client context (e.g., local session), making it harder to scale and recover from failures.

By assuming everything will fail eventually. Use timeouts, circuit breakers, retries with jitter, graceful degradation, and automate failover and recovery processes.

Designing a system so that if a non-critical component fails, the system remains functional but with reduced features (e.g., if the 'recommendations' service is down, still allow the user to checkout).

Isolating elements of an application into pools so that if one fails, the others will continue to function. Named after the partitions in a ship's hull.

Setting strict time limits on calls to external dependencies to prevent a slow service from causing a cascade of waiting threads that eventually crashes the calling service.

An operation is idempotent if it can be performed multiple times without changing the result beyond the initial application. It is crucial for safe retries in distributed systems.

A consistency model used in distributed systems where, if no new updates are made to a data item, all accesses to that item will eventually return the last updated value.

States that a distributed system can only provide two of three guarantees: Consistency, Availability, and Partition Tolerance. In a network partition, SREs must choose between Consistency and Availability.

Identify SPOFs through architectural review and eliminate them by adding redundancy, using load balancers, and implementing distributed consensus algorithms.

A set of policies and procedures to enable the recovery or continuation of vital technology infrastructure and systems following a natural or human-induced disaster.

The maximum acceptable amount of time to restore a system after a failure (how fast must we recover?).

The maximum acceptable amount of data loss measured in time (how much data can we afford to lose?).

The process of determining the resources (CPU, RAM, Storage, Network) needed by a service to meet current and future demand while staying within SLO targets.

By analyzing historical trends, business growth projections (e.g., marketing campaigns), and performing load tests to see how the system behaves under increased pressure.

Load is the amount of work being requested of the system. Capacity is the maximum amount of work the system can handle before performance degrades or it fails.

The difference between your current peak load and your total capacity. Headroom allows you to handle unexpected traffic spikes or component failures without an outage.

Utilization is the average percentage of a resource being used (e.g., 70% CPU). Saturation is the point where the resource has more work than it can handle, leading to queueing (e.g., CPU run queue).

Through Load Testing. You gradually increase traffic until the system hits a 'breaking point' where latency spikes or error rates increase, identifying the maximum stable throughput.

The general practice of testing a system's responsiveness, stability, and speed under a particular workload.

Load Testing: Testing behavior under expected peak load. Stress Testing: Testing behavior beyond normal limits until the system breaks to see how it fails.

Testing the system's reaction to sudden, massive increases in load (e.g., a flash sale) to see if autoscaling and buffers react fast enough.

Running a high load for an extended period (hours or days) to identify long-term issues like memory leaks or disk fragmentation.

JMeter (Java-based, UI/CLI), Gatling (Scala-based, high performance), and Locust (Python-based, easy to script complex scenarios).

The maximum amount of time allowed for each component in a request chain. If an end-to-end request must take < 500ms, the DB might have a budget of 50ms, and the app logic 100ms.

The number of successful actions completed by a system per unit of time (e.g., Requests Per Second or Transactions Per Second).

Latency is time (how long one request takes). Throughput is volume (how many requests we handle at once). They are related but can be optimized independently.

The mathematical study of waiting lines. SREs use it to understand how utilization impacts latency—as utilization nears 100%, waiting times (latency) grow exponentially.

The principle that the average number of items in a stable system ($L$) is equal to the average arrival rate ($lambda$) multiplied by the average time spent in the system ($W$). Formula: $L = lambda W$.

Indexing, query optimization, connection pooling, caching (Redis), read replicas, and horizontal scaling (sharding).

Using efficient data formats (Protobuf/gRPC), minimizing payload sizes, implementing server-side caching, and reducing database round-trips.

A cache of database connections maintained so that connections can be reused when future requests to the database are required, reducing the overhead of opening new connections.

The process of rewriting queries or modifying the DB schema (e.g., adding indices) to ensure that the database can retrieve data using the most efficient path possible.

Carefully selecting which columns to index to speed up reads, while balancing the fact that too many indices can slow down write performance.

A performance bug where an application makes one query to get a list of items, and then one additional query for *each* item in that list, causing massive overhead.

Using the USE Method, looking at profiling data, analyzing distributed traces, and comparing performance metrics against known baselines.

Using specialized tools (e.g., pprof, flame graphs) to measure the execution time and resource usage of specific code paths within an application at runtime.

A methodology for identifying performance issues by checking three metrics for every resource: Utilization (% busy), Saturation (waiting work), and Errors (count).

Kubernetes is a container orchestration platform that provides SREs with the primitive tools to build 'self-healing' infrastructure. It automates deployment, scaling, and management, allowing SREs to define the 'desired state' of an application while the system handles the 'reconciliation loop' to maintain that state.

Key metrics include: Pod status (Pending/Running/Failed), Container CPU/Memory usage vs. Limits, Node status (Ready/NotReady), API Server latency, and Kubelet volume operations.

A state where a Pod starts, crashes, and Kubernetes attempts to restart it repeatedly with an increasing delay (backoff). Usually caused by application errors, missing config files, or failed database connections.

Exit code 137. It occurs when a container exceeds its defined memory limit or the node runs out of memory, and the Linux kernel's Out-Of-Memory killer terminates the process to save the system.

I run `kubectl describe pod`. Common causes include: Insufficient CPU/Memory on nodes, Taints/Tolerations preventing scheduling, or Persistent Volume Claims (PVC) that can't be bound.

Requests: The minimum resources guaranteed to a container (used by the scheduler to place the pod). Limits: The maximum resources a container is allowed to consume. CPU is throttled at the limit; Memory results in an OOMKill.

A controller that automatically scales the number of Pods in a deployment based on observed CPU utilization or custom metrics like request rate.

A tool that automatically sets resource requests and limits for containers based on historical usage, helping to optimize the size of the pods rather than the count.

A component that automatically increases or decreases the size of a Kubernetes cluster (adding/removing nodes) based on the presence of unschedulable pods or underutilized nodes.

Liveness: Checks if the app is alive; if it fails, K8s restarts the container. Readiness: Checks if the app is ready to serve traffic; if it fails, the Pod is removed from the Load Balancer/Service endpoints.

I use `initialDelaySeconds` to let the app start, `periodSeconds` for frequency, and `failureThreshold` to avoid restarts during minor network blips.

A policy that ensures a minimum number of pods remain available during voluntary disruptions (like node maintenance), preventing outages during cluster upgrades.

Rules that constrain which nodes a Pod can be scheduled on. Node Affinity: 'Run this on nodes with SSDs.' Pod Affinity: 'Run this Pod near the Redis Pod for lower latency.'

Taints are applied to Nodes to repel Pods. Tolerations are applied to Pods to allow them to schedule on tainted nodes (e.g., keeping production pods off GPU nodes unless required).

By setting `maxUnavailable` and `maxSurge` in the deployment strategy. This ensures that a specific number of old pods stay alive while new ones are verified by readiness probes.

RollingUpdate: Gradually replaces pods (zero downtime). Recreate: Kills all old pods first, then starts new ones (causes downtime, but prevents version mismatch).

Using `kubectl rollout undo deployment/name`. This reverts the deployment to the previous revision in the history.

Used for applications that require stable identifiers (pod-0, pod-1) and persistent storage (like Databases or Kafka), where the order of startup and shutdown matters.

Ensures that a copy of a specific Pod runs on every node in the cluster. Typical use cases include log collectors (Fluentd) and monitoring agents (Node Exporter).

Resource Quota: Restricts total resource usage (CPU/RAM) for a whole Namespace. Limit Range: Sets default or min/max resource constraints for individual containers in a Namespace.

I use Prometheus with kube-state-metrics (for object status) and node-exporter (for hardware metrics), combined with Grafana dashboards for cluster-wide visibility.

A service that listens to the Kubernetes API server and generates metrics about the state of objects like deployments, nodes, and pods (e.g., 'Is this pod running?').

A Prometheus agent that runs on every node to collect hardware and OS-level metrics (CPU usage, Disk I/O, Network traffic).

A command that provides a real-time view of CPU and Memory consumption for nodes or pods, similar to the `top` command in Linux.

1. `kubectl logs`. 2. `kubectl describe`. 3. `kubectl exec -it -- /bin/bash` (to get a shell). 4. `kubectl get events` to check for scheduling or image pull failures.

A special type of container that you can 'inject' into an existing Pod for debugging purposes when the main container lacks troubleshooting tools (like `curl` or `dig`).

Objects that record state changes and errors (e.g., NodeUnreachable, PulledImage). Monitoring these is crucial for catching transient issues that don't always trigger an alert.

Using PersistentVolumes (PV) and PersistentVolumeClaims (PVC). The PVC allows the app to request storage, and the StorageClass handles the automatic provisioning of the actual disk on the cloud provider.

A way for administrators to describe the 'classes' of storage offered (e.g., 'fast-ssd' vs 'slow-hdd'). It enables dynamic provisioning of persistent volumes.

A plugin-based system that handles the networking between Pods and Nodes. Examples include Calico, Flannel, and Cilium.

I use the `uptime` or `top` command. It shows the average number of processes in a runnable or uninterruptible state over 1, 5, and 15-minute intervals.

They represent exponential moving averages. If the number exceeds the CPU core count, it indicates CPU saturation and that processes are waiting for CPU time.

I use `top` or `htop` and sort by `%CPU`. To see historical data, I might use `pidstat` or check Prometheus metrics.

Using `free -m` for system-wide stats and `top` (sorted by `%MEM`) for process-specific usage. I also check 'Available' memory rather than just 'Free'.

RAM is fast physical memory. Swap is a portion of the hard drive used when RAM is full. Excessive swapping ('thrashing') significantly degrades performance.

The Out-Of-Memory Killer is a Linux kernel feature that terminates processes to prevent the entire system from crashing when memory is exhausted. It targets processes with high 'oom_score'.

1. `df -h` to see which partition is full. 2. `du -sh *` to find large directories. 3. `lsof | grep deleted` to find files that are deleted but still held open by a process.

An inode is a data structure on a Linux filesystem that stores metadata about a file. You run out of inodes if you have millions of tiny files, even if you have plenty of actual disk space remaining.

I use the `lsof` (List Open Files) command or `fuser`. For example: `lsof /path/to/file`.

A powerful utility that lists all open files and the processes that opened them, including network sockets and pipes.

Using `ss` (Socket Statistics) or `netstat`. `ss -tunlp` is commonly used to see TCP/UDP connections and the associated process IDs.

`ss` is the modern, faster replacement for `netstat`. Both are used to investigate established connections, listening ports, and routing tables.

Using `dig`, `nslookup`, or `host`. I check `/etc/resolv.conf` for nameservers and use `dig +trace` to follow the resolution path.

I use `tcpdump` on the CLI to capture traffic (e.g., `tcpdump -i eth0 port 80`). I then export the `.pcap` file to Wireshark for visual analysis.

A diagnostic tool that records all system calls made by a process. I use it when an app is hanging or failing and I need to see what file or network operation it's stuck on.

The programmatic way a program requests a service from the kernel (e.g., `open`, `read`, `write`, `network`).

Using `ps -eLf` or `top -H`. This shows individual threads as separate lines, helping identify multi-threaded performance issues.

The process of the CPU storing the state of one process/thread so that it can restore and run another. High context switching indicates a system is overloaded.

Soft Limit: The value the kernel enforces for a resource (like open files). Hard Limit: The ceiling for the soft limit that a non-root user cannot exceed.

Using the `sysctl` command or editing `/etc/sysctl.conf`. Common SRE tuning includes increasing the maximum number of open files or adjusting TCP buffer sizes.

A file descriptor is an index to an entry in the kernel-resident data structure containing the details of all open files. In SRE, we increase the limits using `ulimit -n` or by modifying `/etc/security/limits.conf` to prevent 'Too many open files' errors in high-concurrency apps.

TIME_WAIT is a state where the socket is kept open for a short period after closing to ensure that any delayed packets are handled. High rates of new connections can lead to 'Socket Exhaustion' if too many sockets stay in this state.

Common states include: LISTEN (waiting), SYN_SENT (initiating), ESTABLISHED (connected), FIN_WAIT (closing), and TIME_WAIT (cooldown).

I use `perf record` to sample the CPU and `perf report` to analyze where the cycles are being spent. It helps identify which function in the kernel or application is causing high CPU usage.

eBPF is a revolutionary technology that allows running sandboxed programs in the Linux kernel without changing kernel source code. SREs use it for high-performance networking, security, and deep observability (e.g., tools like Cilium or Pixie).

By identifying repetitive manual tasks (like server provisioning or log rotation) and writing scripts or using orchestration tools (Terraform/Ansible) to handle them. The goal is to make the operation self-service or autonomous.

Tools used to maintain the 'state' of a server. They ensure that the correct packages, files, and users are present. Ansible is agentless and uses YAML, while Puppet and Chef usually use agents.

Defining and provisioning data center infrastructure using machine-readable definition files. This allows for version control, peer reviews, and consistent environments across dev/prod.

Imperative defines 'how' to do it (step-by-step scripts). Declarative defines 'what' the end state should look like (e.g., 'I want 3 servers'), and the tool figures out how to achieve it.

An operational framework where Git is the 'Single Source of Truth' for infrastructure and applications. Changes are made via Pull Requests, and a controller (like ArgoCD) syncs the live state to the Git state.

A continuous process in Kubernetes and GitOps where the system compares the 'Desired State' (Git/YAML) with the 'Actual State' (Live cluster) and takes action to correct any drift.

Never hardcode secrets. I use tools like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets in Kubernetes to inject sensitive data into the environment at runtime.

The property where running an automation script multiple times produces the same result as running it once, without causing errors or unwanted side effects.

Using linters, 'plan' stages (like `terraform plan`), and spinning up ephemeral environments (using Terratest or Kitchen-CI) to verify changes before they hit production.

Using tools like Open Policy Agent (OPA) to enforce rules on infrastructure (e.g., 'No public S3 buckets' or 'All pods must have resource limits').

Delivery: Code is automatically built and tested, but manual approval is needed to deploy to prod. Deployment: Every change that passes tests is automatically pushed to production.

By using techniques like Canary releases, Blue-Green deployments, and automated health checks that trigger a rollback if error rates spike during a rollout.

A strategy where a new version of software is deployed to a small percentage of users first. If the metrics look good, it's rolled out to everyone. If not, the 'Canary' is killed.

Having two identical environments. 'Blue' is live, and you deploy to 'Green'. Once tested, you switch the Load Balancer to 'Green'. If a bug is found, you switch back to 'Blue' instantly.

A software toggle that allows you to turn features on or off at runtime without a new deployment. SREs use them to 'kill' a buggy feature if it causes instability.

By setting up monitoring alerts that, when triggered during a deployment, automatically execute a script to revert the deployment to the previous stable version.

An evolution of Continuous Delivery that includes advanced deployment patterns like traffic shadowing and canary rollouts based on real-time observability data.

The automated sequence of steps (Build, Test, Scan, Deploy) that code goes through to move from a developer's machine to the production environment.

By using 'Auto-Remediation' tools. For example, if a service is OOM-ing, the system automatically triggers a script to increase memory limits or restart the pod.

Integrating operational tools into a chat platform (Slack/Teams). SREs can trigger deployments, check status, or ack alerts directly from the chat window.

I track Query Latency, Connection Pool usage, Disk I/O, CPU, and specifically Slow Queries. I also monitor replication lag to ensure data consistency.

When an application requests more connections to the database than the DB or the pooler can handle. This causes new requests to wait or fail, spiking application latency.

I use `EXPLAIN ANALYZE` to see the query execution plan. I look for missing indexes, full table scans, or complex joins that can be optimized or cached.

The sequence of steps a database engine takes to execute a query. Understanding this is key to performance tuning.

By using a 'Leader/Follower' setup. If the leader fails, a follower is promoted to leader. Tools like Patroni or cloud-native RDS handles this automatically.

The time delay between when a write happens on the Primary database and when that change appears on the Replica. High lag causes 'stale' data reads.

A condition where two nodes in a cluster both believe they are the 'Leader'. This leads to data corruption as both nodes accept writes independently.

I use a combination of Full Backups (Snapshot) and Point-In-Time Recovery (PITR) using WAL (Write Ahead Logs) to ensure minimal data loss.

The ability to restore a database to its exact state at a specific second in the past by replaying transaction logs over a full backup.

A backup is only good if it can be restored. SREs automate a 'Restore Drill' where a backup is restored to a temporary instance and verified for data integrity weekly.

I have extensive experience with AWS, Azure, and GCP. My core focus is on leveraging cloud-native services like EKS/GKE for orchestration, RDS for managed databases, and S3/Blob Storage for scalable object storage while maintaining infrastructure as code using Terraform.

Reliability in the cloud is achieved through multi-layered redundancy: deploying across multiple Availability Zones (AZs) to protect against data center failure, using Global Load Balancers for multi-region failover, and implementing auto-scaling groups to handle traffic spikes and replace unhealthy instances automatically.

Multi-AZ (Availability Zone) deployment involves running your application and database across multiple isolated data centers within a single region. This ensures that if one data center loses power or connectivity, the service remains available via the others.

Deploying an application across different geographical areas (e.g., US-East-1 and EU-West-1). This protects against total cloud provider regional outages and reduces latency for global users by serving them from the nearest location.

By designing for 'Region Evacuation.' We use global DNS (like Route53) to shift traffic away from the failing region, maintain synchronized database replicas in a secondary region, and ensure our CI/CD pipelines can deploy the stack to a different region or even a different cloud provider (multi-cloud strategy).

A service that monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost, scaling instances up during high demand and down during low demand.

Elastic Load Balancing health checks periodically send requests to registered targets to test their status. If a target is found unhealthy, the load balancer stops routing traffic to it until it passes the checks again.

I use a combination of cloud-native tools (AWS CloudWatch, Azure Monitor) for infrastructure-level metrics and external agents (Prometheus, Datadog) for application-level metrics and custom SLIs.

CloudWatch is integrated, requires zero setup for AWS services, but can be expensive and has higher latency. Prometheus offers better query flexibility (PromQL), lower latency, and is cloud-agnostic, but requires manual management of the monitoring infra.

A system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to achieve a common goal as a single coherent system.

The main challenges are Network Partitioning (Partial failure), Latency, Clock Skew (unsynchronized time), Data Consistency (CAP theorem), and Service Discovery.

A failure in the network that causes members of a distributed system to split into two or more groups that cannot communicate with each other, though each group remains internally functional.

A condition in a high-availability cluster where two nodes simultaneously believe they are the 'Leader' due to a loss of communication, leading to data corruption as both write to the shared storage or database.

By designing applications to be idempotent and using versioning/timestamps. We accept that data might be slightly stale for a few milliseconds but ensure that the system eventually converges to a single correct state.

Algorithms used to ensure that all nodes in a distributed system agree on a single data value or state, even if some nodes fail. Raft is commonly used in tools like etcd (the heart of Kubernetes).

An infrastructure layer that handles service-to-service communication, providing features like traffic management, mTLS security, and deep observability (tracing/metrics) without modifying application code.

A design pattern where an auxiliary container (the sidecar) runs alongside the main application container in a Pod. The sidecar handles cross-cutting concerns like logging, proxying, or secret injection.

A single entry point for all clients that routes requests to various backend microservices, often handling authentication, rate limiting, and request transformation.

A mechanism where a downstream service tells an upstream service to slow down because it is overwhelmed, preventing the entire system from crashing due to resource exhaustion.

By implementing Circuit Breakers to cut off failing dependencies, Timeouts to prevent hanging connections, and Exponential Backoff with jitter for retries to avoid 'thundering herd' spikes.

Partitioning resources (e.g., thread pools or server groups) so that a failure in one service or component doesn't consume all system resources and cause the entire application to fail.

The practice of passing the remaining 'time budget' for a request down the call chain. If the top-level request has 500ms left, the database call at the bottom should know it only has 500ms (or less) to respond.

It is vital for understanding latency bottlenecks and debugging errors that span multiple services, allowing SREs to see a 'waterfall' view of exactly where a request spent its time.

I rely on the 'Three Pillars': correlating high-level metrics (latency spikes) with specific traces (bottlenecked service) and then deep-diving into structured logs for the specific Request-ID.

The ability to ask arbitrary questions about your system without knowing the failure modes in advance. It requires deep telemetry that reveals the interactions between decoupled components.

In SRE, security is part of reliability. A system that is breached or defaced is not reliable. SREs focus on automating security patches, managing secrets securely, and ensuring that security controls don't break system availability.

By using IAM roles and Kubernetes RBAC to grant users and services only the permissions they need to function. For example, a pod reading from S3 should only have 'GetObject' permissions, not 'Delete' or 'List'.

The process of storing and managing sensitive data (API keys, passwords) using specialized tools like HashiCorp Vault or AWS Secrets Manager rather than environment variables or Git.

The automated process of replacing SSL/TLS certificates before they expire. SREs use tools like cert-manager in K8s to handle this to avoid the 'Expired Cert' outages common in manual environments.

By following the standard incident response process: Detect (SIEM alerts), Contain (isolate compromised pods/keys), Eradicate (patch/fix), and Recover, followed by a blameless postmortem.

Using Web Application Firewalls (WAF), Anycast networks (CDNs), and rate-limiting at the edge to absorb and drop malicious traffic before it reaches your backend servers.

A defense against brute-force attacks and resource exhaustion, where we limit the frequency of requests from a single IP or API key to prevent a malicious actor from overwhelming the service.

By monitoring for anomalies like sudden spikes in 401/403 errors, unusual outbound traffic to unknown IPs, and auditing Kubernetes API server logs for unauthorized access attempts.

Maintaining an immutable record of 'Who did what, when, and where' across the infrastructure. This is critical for both forensic investigations after a breach and for regulatory compliance.

Continuous checks to ensure that the live infrastructure matches the required security standards (e.g., SOC2, PCI-DSS) using tools like AWS Config or Open Policy Agent.

By automating the image building process. When a CVE is found in a base image, the pipeline automatically rebuilds the app with the patched image and performs a rolling update to production.

A risk-based approach where 'Critical' and 'High' vulnerabilities are patched immediately, while others are grouped into regular maintenance cycles, using blue-green deployments to minimize impact.

Using SAST (Static Analysis) for code, DAST (Dynamic Analysis) for running apps, and SCA (Software Composition Analysis) for third-party libraries within the build pipeline.

By enforcing TLS for all data in transit (mTLS within the mesh) and using AES-256 for data at rest (S3 encryption, EBS volume encryption) with keys managed by a KMS.

A multi-layered security approach. If one defense (like a firewall) fails, others (like mTLS, IAM roles, and disk encryption) are in place to prevent a total breach.

1. Check the Global Load Balancer metrics for latency spikes. 2. Look at the Golden Signals (Latency, Traffic, Errors, Saturation). 3. Use distributed tracing to find the specific slow service. 4. Check that service's logs and resource usage (CPU/RAM/DB locks).

1. Confirm the outage via health checks. 2. Initiate failover to the standby replica if not automatic. 3. Update the Status Page. 4. Investigate logs to find if it was a crash, resource exhaustion, or hardware failure while the standby handles traffic.

1. Run `top` or `htop` to identify the specific process. 2. Check if the spike correlates with a traffic increase or a recent deployment. 3. Profile the process (e.g., `perf` or `pprof`) to find the code path consuming cycles. 4. Scale out or rollback if necessary.

1. Check for OOMKiller events in dmesg. 2. Identify if there is a slow leak using memory profiling tools. 3. Temporarily increase memory limits or scale horizontally to keep the service alive while investigating the root cause.

1. Identify large files using `du`. 2. Clear out temp files/old logs. 3. Check for 'deleted' files still held by processes using `lsof`. 4. Expand the volume size or implement better log rotation/retention policies.

1. `kubectl logs --previous` to see why the last instance failed. 2. `kubectl describe pod` to check for health check failures or configuration errors. 3. Check for missing secrets or environment variables that the app needs to start.

1. Verify the app's health check endpoint manually (curl). 2. Check security groups/firewalls to see if the LB can reach the targets. 3. Review app logs for startup failures or connection timeouts to the database.

1. Determine if it's network-wide or service-specific. 2. Check for resource saturation (CPU/IO). 3. Look for 'Stop the World' Garbage Collection events in the app logs. 4. Check for downstream dependency slowdowns via tracing.

1. Check the status code distribution (5xx vs 4xx). 2. Correlate with recent changes (deploys/config updates). 3. Rollback the latest change if it correlates. 4. Deep-dive into logs for stack traces or error messages.

1. Check DNS resolution and record validity. 2. Inspect Edge/CDN and WAF logs for accidental blocks. 3. Check Global Load Balancer health. 4. Verify that the ISP/Cloud Provider isn't having a backbone outage.

If using Kubernetes, I run `kubectl rollout undo`. If using a Blue-Green strategy, I switch the Load Balancer traffic back to the 'Blue' environment. For Terraform, I revert the Git commit and re-apply the previous state. I always verify the health of the system immediately after the rollback.

1. Check the network throughput between primary and replica. 2. Look for long-running transactions or heavy write load on the primary. 3. Check for resource contention (CPU/Disk IO) on the replica. 4. If using Postgres, check `pg_stat_replication`. Consider temporary vertical scaling of the replica.

1. Identify if it's a spike or a slow leak using Grafana. 2. Check if the application is hitting its cgroup limits. 3. Take a heap dump or use a profiler to identify memory-hungry objects. 4. If the service is critical, increase the 'limit' temporarily while a permanent fix is developed.

1. Renew the certificate immediately via the CA. 2. Manually update the Load Balancer or Secret. 3. Investigate why the automation (e.g., cert-manager) failed. 4. Add a monitoring alert that triggers 30 days before the next expiration.

1. Use `dig` or `nslookup` to test locally vs. against a public DNS (8.8.8.8). 2. Check for expired domain registrations. 3. Verify DNS records in the cloud console. 4. Check for VPC-local DNS service outages or `/etc/resolv.conf` misconfigurations.

1. Check application logs for stack traces. 2. Look at upstream and downstream metrics to identify if it's a code error or a dependency failure. 3. Check for recent config changes. 4. If it's a sudden spike across all nodes, it's likely a global state or DB issue.

1. Check if the consumer processes are still running. 2. Look for high processing time per message. 3. Check for 'rebalancing' issues. 4. Increase the number of partitions or consumers to scale throughput if the load has permanently increased.

1. Check if the cache was flushed recently. 2. Look for changes in data access patterns or key expiration (TTL) settings. 3. Check if Redis has hit its memory limit and is evicting keys (check eviction policy).

1. Use tools like `mtr` or `ping` to check for packet loss. 2. Check for service mesh overhead or mTLS handshake delays. 3. Verify if services are in the same AZ/Region. 4. Look for saturated NAT gateways or bandwidth throttling at the cloud level.

1. Use Cost Explorer to identify the specific service (e.g., S3, Data Transfer). 2. Correlate with recent architectural changes or traffic spikes. 3. Look for orphaned resources (unattached EBS volumes) or inefficient scaling policies.

I'd start with the Four Golden Signals. I'd define SLIs for core functionality, set SLOs based on business needs, instrument the code with Prometheus exporters, and build a Grafana dashboard that visualizes the error budget.

By using a RollingUpdate strategy in K8s with proper Readiness Probes. This ensures new versions are only given traffic after they pass health checks, and old versions are only shut down after new ones are ready.

By executing the Disaster Recovery plan. I'd update DNS to point to the healthy secondary region, verify that database failover has completed (RPO check), and scale up resources in the active region to handle the consolidated load.

I'd use the Expand and Contract pattern. 1. Deploy the new service alongside the old one. 2. Double-write or sync data. 3. Gradually shift a percentage of traffic (Canary). 4. Decommission the old service once 100% of traffic is stable on the new one.

1. Pre-scale all clusters. 2. Enable 'Feature Flags' to turn off non-essential services (e.g., reviews). 3. Implement aggressive caching. 4. Freeze all code deployments one week prior. 5. Monitor the 'Traffic' golden signal closely.

I would establish a 'Pilot Light' or 'Warm Standby' architecture in a different region. I'd use automated Terraform scripts to spin up the full stack and regular RPO/RTO drills to ensure data is actually recoverable.

1. Parallelize CI tests. 2. Optimize Docker image layers to reduce pull time. 3. Use 'Skaffold' or similar for faster local feedback. 4. Implement incremental builds and only re-test changed modules.

By improving 'Observability' to reduce detection time, creating high-quality 'Runbooks' for common issues, and automating common remediation tasks like restarts or traffic shifting.

1. Identify the process using `top`. 2. Use `strace` or `lsof` to see what it's doing. 3. If it's a bug, kill it and let the scheduler (K8s) restart it. 4. Apply resource limits (cgroups) to prevent it from affecting neighboring services.

These are the hardest. I would look for 'Micro-bursts' in traffic, check TCP connection pools, look at GC pauses in the app logs, and use distributed tracing to see if a specific downstream hop is occasionally slow.

1. Load Balancer target group status. 2. Service mesh routing rules (VirtualServices in Istio). 3. DNS records. 4. WAF rules that might be blocking the source traffic.

I'd use a profiling tool (like JProfiler, Go pprof, or valgrind) to compare memory snapshots over time. I'd look for objects that grow indefinitely and are never garbage collected.

1. Temporarily increase the DB max connection limit if possible. 2. Check for 'Leaking' connections in the application. 3. Implement or tune a Connection Pooler (like PgBouncer). 4. Scale the application down if it's overwhelming the DB.

Implement the Circuit Breaker pattern to stop calling the API and failing fast. If possible, provide a cached result or a fallback experience to the user so they don't see a 500 error.

I'd focus on Availability (successful transactions) and Latency (time to process). I'd set a target like '99.95% of payments succeed' and '99% of payments complete within 2 seconds.'

By identifying the most frequent manual task (e.g., user onboarding) and writing a script or using a tool to make it self-service. I'd document the process so others can contribute to the automation.

I'd prioritize based on Severity and Impact. The outage affecting the most users or core business (e.g., checkout) gets the primary focus. I'd delegate the second incident to a secondary responder or another team.

1. Define core metrics. 2. Add Prometheus annotations or configuration. 3. Create a standard dashboard template. 4. Set up alerts for SLO violations. 5. Verify the logging and tracing integration.

I'd start small in a staging environment using a tool like Chaos Mesh or AWS Fault Injection. I'd terminate a single pod or inject latency, then verify that the monitoring detects it and the system recovers automatically.

1. Add structured logging if possible. 2. Deploy a sidecar agent to collect hardware/network metrics. 3. Implement an 'External Prober' to check availability from the outside-in. 4. Use eBPF-based tools to observe system calls without changing code.

Focus on the S.T.A.R. method. Describe the Situation (outage), Task (mitigate), Action (steps taken, communication), and Result (lessons learned, future prevention). Emphasize the 'blameless' approach.

I focus on Mitigation first ('stopping the bleeding') to restore service to users. Technical deep-diving for the root cause happens only after the system is stable.

Example: Noticing a trend in resource usage during a load test and scaling the DB before a marketing campaign launched. This shows proactive 'Engineering time' value.

I stay calm by following the established incident response process. I focus on clear communication and data-driven decisions rather than guessing.

Using the Error Budget. If the budget is full, we move fast. If it's empty, we prioritize reliability. This removes the 'argument' between Dev and SRE.

1. Compare metrics between the current and previous versions (A/B dashboard). 2. Look for increased CPU/GC time. 3. Use distributed tracing to see if a specific code path got slower. 4. If unclear, rollback and profile the new version in a staging environment with production-like load.