Backend Developer

Backend development involves server-side programming that powers the logic, database interactions, authentication, and server configuration of web applications. Core responsibilities include: API development, database management, server logic implementation, authentication/authorization, data validation, business logic processing, integration with third-party services, performance optimization, and security implementation. The backend serves as the brain of the application, processing requests from the frontend and returning appropriate responses.

Frontend (client-side) is what users see and interact with directly - HTML, CSS, JavaScript, UI/UX. It runs in the browser. Backend (server-side) handles business logic, database operations, authentication, and API endpoints. It runs on servers. Example: In an e-commerce app, the product display page is frontend, while the payment processing, inventory management, and user authentication are backend operations.

Client-Server architecture is a distributed computing model where clients (browsers, mobile apps) request services and servers provide them. Flow: 1) Client sends HTTP request to server, 2) Server processes request (queries DB, runs business logic), 3) Server sends HTTP response back to client, 4) Client renders the data. Advantages: Centralized data management, better security, scalability, and separation of concerns.

API (Application Programming Interface) is a set of rules and protocols that allows different software applications to communicate with each other. It defines endpoints, request/response formats, and authentication methods. Importance: Enables frontend-backend communication, third-party integrations (payment gateways, maps), microservices communication, and platform independence. Example: Weather apps use weather API to fetch current temperature data.

Monolithic: Single unified codebase where all components (UI, business logic, DB access) are tightly coupled and deployed together. Pros: Simple development, easy debugging. Cons: Scaling issues, deployment complexity. Microservices: Application broken into small, independent services communicating via APIs. Each service has its own database and can be deployed independently. Pros: Better scalability, technology flexibility, fault isolation. Cons: Complex management, network overhead. Example: Netflix uses microservices where recommendation, streaming, and billing are separate services.

Stateless: Each request is independent, server doesn't store client state between requests. Session data stored client-side (JWT) or external store (Redis). Benefits: Easy horizontal scaling, fault tolerance, simpler deployment. Example: REST APIs with JWT. Stateful: Server maintains session state. Example: Traditional session-based auth (session ID in cookie, data on server). Benefits: Better security control, easier to invalidate sessions. Cons: Harder to scale (need sticky sessions or shared session store). Modern apps prefer stateless for scalability, use external cache for state if needed. Example: User cart stored in Redis (stateless app, external state).

Synchronous: Operations execute sequentially, one after another. Current operation must complete before next starts. Blocks execution. Example: const data = fs.readFileSync('file.txt'); console.log(data); console.log('done'); // Waits for file read. Asynchronous: Non-blocking. Operation starts, control returns immediately, callback/promise executes when operation completes. Example: fs.readFile('file.txt', (err, data) => console.log(data)); console.log('done'); // Executes immediately, before file read completes. Node.js is async by default for I/O. Benefits: Better performance (don't wait for slow I/O), handle many requests concurrently. Use callbacks, promises, async/await. Example async/await: const data = await fs.promises.readFile('file.txt'); console.log(data);

REST (Representational State Transfer) is an architectural style for designing networked applications. RESTful principles: 1) Stateless - each request contains all info needed, 2) Client-Server separation, 3) Cacheable responses, 4) Uniform Interface - consistent naming conventions, 5) Layered System - client doesn't know if connected to end server or intermediary, 6) Resource-based URLs (/users/123). Uses standard HTTP methods: GET (read), POST (create), PUT (update), DELETE (remove).

GET: Retrieve data. Example: GET /api/users/5 - fetch user with ID 5. Idempotent & safe. POST: Create new resource. Example: POST /api/users {name: 'John'} - create new user. Not idempotent. PUT: Replace entire resource. Example: PUT /api/users/5 {name: 'Jane', email: 'jane@email.com'} - replace user 5. Idempotent. PATCH: Partial update. Example: PATCH /api/users/5 {name: 'Jane'} - update only name. DELETE: Remove resource. Example: DELETE /api/users/5 - delete user 5. Idempotent.

HTTP status codes indicate the result of an HTTP request. 1xx (Informational): 100 Continue. 2xx (Success): 200 OK, 201 Created, 204 No Content. 3xx (Redirection): 301 Moved Permanently, 304 Not Modified. 4xx (Client Error): 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 422 Unprocessable Entity, 429 Too Many Requests. 5xx (Server Error): 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout.

PUT replaces the entire resource with the provided data. If you send PUT /users/1 {name: 'John'}, it replaces the whole user object, potentially removing other fields. PATCH performs partial updates, modifying only specified fields. PATCH /users/1 {name: 'John'} updates only the name field while preserving other fields like email, age, etc. Best practice: Use PATCH for partial updates to avoid accidentally deleting data.

API versioning allows you to make changes to your API without breaking existing clients. Methods: 1) URL versioning: /api/v1/users, /api/v2/users, 2) Header versioning: Accept: application/vnd.company.v2+json, 3) Query parameter: /api/users?version=2. Importance: Backward compatibility, smooth transitions, allows deprecated feature removal, supports multiple client versions simultaneously. Example: Twitter API v1.1 vs v2 - different authentication and response formats.

Pagination limits the number of results returned per request to improve performance. Implementation methods: 1) Offset-based: /api/users?limit=20&offset=40 (skip first 40, return next 20), 2) Page-based: /api/users?page=3&size=20, 3) Cursor-based: /api/users?cursor=xyz123 (uses unique identifier, best for real-time data). Response should include metadata: {data: [...], total: 1000, page: 3, hasNext: true}. Cursor-based is preferred for large datasets as it handles new insertions better.

CORS (Cross-Origin Resource Sharing) is a security mechanism that restricts web pages from making requests to a different domain than the one serving the page. Browser blocks requests unless server explicitly allows it. Solution: Server must include CORS headers: Access-Control-Allow-Origin: https://frontend.com, Access-Control-Allow-Methods: GET, POST, PUT, Access-Control-Allow-Headers: Content-Type, Authorization. In Express.js: Use cors middleware: app.use(cors({origin: 'https://frontend.com', credentials: true})).

REST: Multiple endpoints for different resources (/users, /posts), over-fetching (getting unnecessary data) or under-fetching (multiple requests needed), versioning required. GraphQL: Single endpoint (/graphql), client specifies exactly what data needed, no over/under-fetching, schema-based with strong typing, real-time updates via subscriptions. Example: REST needs 3 requests (user, posts, comments) while GraphQL can fetch all in one query. Use REST for simple CRUD apps, GraphQL for complex data requirements with multiple relationships.

Rate limiting restricts the number of API requests a client can make in a given time period to prevent abuse and ensure fair usage. Implementation: 1) Fixed Window: 100 requests/hour, counter resets every hour, 2) Sliding Window: More accurate, tracks requests in rolling time window, 3) Token Bucket: Tokens refill at fixed rate, request consumes token. Use Redis for distributed rate limiting. Express.js example: const rateLimit = require('express-rate-limit'); const limiter = rateLimit({windowMs: 15*60*1000, max: 100}); app.use('/api/', limiter);. Return 429 status when limit exceeded.

API authentication verifies the identity of the client making requests. Methods: 1) Basic Auth: Base64 encoded username:password in header (insecure without HTTPS), 2) API Keys: Unique key per client (simple but less secure), 3) OAuth 2.0: Token-based, supports delegated access (Google Sign-In), 4) JWT (JSON Web Tokens): Stateless, self-contained tokens with user info and signature, 5) Session-based: Server stores session, client sends session ID in cookie. Best practice: Use JWT for stateless APIs, OAuth for third-party integrations.

SQL (Relational): Structured schema with tables, rows, columns. ACID properties (Atomicity, Consistency, Isolation, Durability). Supports complex queries and joins. Vertical scaling. Examples: MySQL, PostgreSQL, Oracle. Use for: Financial systems, transactional apps. NoSQL (Non-relational): Flexible schema, various types (document, key-value, graph, column-family). Eventual consistency. Horizontal scaling. Examples: MongoDB, Redis, Cassandra. Use for: Big data, real-time apps, unstructured data. Example: E-commerce uses SQL for orders, NoSQL for product catalog.

Normalization eliminates data redundancy and improves integrity. Forms: 1NF: Atomic values (no multi-value attributes), each column has unique name. 2NF: Meets 1NF + no partial dependencies (non-key attributes depend on entire primary key). 3NF: Meets 2NF + no transitive dependencies (non-key attributes depend only on primary key). BCNF: Stricter 3NF. Denormalization: Intentionally adding redundancy for performance (read-heavy systems). Example: Instead of storing user address in orders table (redundant), create separate users table and reference user_id.

Index is a data structure (usually B-Tree) that improves query speed by creating pointers to data locations. Without index: Full table scan O(n). With index: Binary search O(log n). Types: 1) Primary Index: On primary key (auto-created), 2) Secondary Index: On non-primary columns, 3) Composite Index: Multiple columns, 4) Unique Index: Ensures uniqueness. Trade-off: Faster reads but slower writes (index must be updated). Example: CREATE INDEX idx_email ON users(email); makes SELECT * FROM users WHERE email='x' much faster. Use on frequently queried columns.

ACID ensures reliable transactions: Atomicity: All operations in transaction succeed or all fail (no partial updates). Example: Money transfer - debit and credit both must happen. Consistency: Database moves from one valid state to another (constraints maintained). Isolation: Concurrent transactions don't interfere. Transaction A can't see uncommitted changes of B. Durability: Committed transactions persist even after system failure (written to disk). SQL databases guarantee ACID. NoSQL often trades ACID for performance (eventual consistency).

Sharding is horizontal partitioning where data is split across multiple database servers (shards) based on a shard key. Example: Users 1-1M on Shard1, 1M-2M on Shard2. Types: 1) Range-based: User IDs 1-1000 → Shard1, 2) Hash-based: hash(user_id) % num_shards, 3) Geographic: US users → US shard, EU → EU shard. Benefits: Improved performance, horizontal scaling. Challenges: Complex queries across shards, rebalancing. Use when: Single DB can't handle load, millions of records. Example: Instagram shards user data by user_id.

Transaction: Group of operations executed as a single unit. Isolation levels (weakest to strongest): 1) Read Uncommitted: Can read uncommitted changes (dirty reads), 2) Read Committed: Only reads committed data (prevents dirty reads), 3) Repeatable Read: Same query returns same results during transaction (prevents non-repeatable reads), 4) Serializable: Strongest, transactions appear sequential (prevents phantom reads). Trade-off: Stronger isolation = better consistency but lower concurrency. Example: BEGIN TRANSACTION; UPDATE accounts SET balance=balance-100 WHERE id=1; UPDATE accounts SET balance=balance+100 WHERE id=2; COMMIT;

N+1 problem occurs when you execute 1 query to fetch N records, then N additional queries to fetch related data. Example: Fetch 100 users (1 query), then for each user fetch their posts (100 queries) = 101 total queries. Solution: 1) Eager Loading/JOIN: Fetch users and posts in single query: SELECT users.*, posts.* FROM users LEFT JOIN posts ON users.id=posts.user_id, 2) DataLoader pattern (batching), 3) Use ORM features: In Sequelize: User.findAll({include: Post}). This reduces 101 queries to 1-2 queries, dramatically improving performance.

Connection pooling maintains a pool of reusable database connections instead of creating/destroying connections for each request. Benefits: Reduced overhead (connection creation is expensive), better performance, controlled resource usage. Configuration: min connections (always maintained), max connections (upper limit), idle timeout. Example in Node.js with pg: const pool = new Pool({max: 20, min: 5, idleTimeoutMillis: 30000}); pool.query('SELECT * FROM users'). Without pooling: Each request creates new connection (slow). With pooling: Reuses existing connections (fast).

Replication copies data from one database (primary/master) to one or more databases (replicas/slaves) for availability and performance. Types: 1) Master-Slave: Primary handles writes, replicas handle reads. If primary fails, promote replica. 2) Master-Master: Multiple primaries, both handle reads/writes (conflict resolution needed). 3) Synchronous: Replica updated before transaction completes (consistent but slow). 4) Asynchronous: Replica updated after (fast but eventual consistency). Use cases: Load distribution (read from replicas), disaster recovery, geographic distribution. Example: MySQL master-slave replication.

Database migrations are version-controlled changes to database schema (create tables, add columns, modify constraints). Benefits: Track schema changes over time, reproducible across environments (dev, staging, prod), team collaboration, rollback capability. Tools: Knex.js, Sequelize, Flyway, Alembic. Example migration: exports.up = function(knex) { return knex.schema.createTable('users', table => { table.increments('id'); table.string('email').unique(); table.timestamps(); }); }; exports.down = function(knex) { return knex.schema.dropTable('users'); }; Run: npx knex migrate:latest.

Connection leak: Opened database connections not properly closed, eventually exhausting connection pool. Symptoms: 'Too many connections' error, app hangs. Causes: 1) Not releasing connections after use, 2) Exceptions before connection.close(), 3) Forgotten transactions. Prevention: 1) Always use try-finally: let conn; try { conn = await pool.getConnection(); // use conn } finally { if(conn) conn.release(); } 2) Use ORM (auto-manages connections), 3) Connection pooling with timeouts, 4) Monitor: Active connections, pool exhaustion alerts. Example: const conn = await pool.getConnection(); await conn.query('SELECT * FROM users'); conn.release(); // CRITICAL! Debugging: Log connection acquisition/release, track connection lifecycle.

Connection pool configuration parameters: 1) min: Minimum connections always maintained (avoid cold start latency), 2) max: Maximum concurrent connections (prevent DB overload), 3) idleTimeoutMillis: Close idle connections after timeout, 4) connectionTimeoutMillis: Max wait time for connection from pool, 5) maxLifetime: Max connection age before refresh. Tuning: Formula: max = ((core_count * 2) + effective_spindle_count). Example: 4 cores, 1 disk = 9-10 connections. Monitor: Pool exhaustion, wait times, idle connections. Example config: {min: 5, max: 20, idleTimeoutMillis: 30000, connectionTimeoutMillis: 5000}. Over-provisioning wastes DB resources. Under-provisioning causes request failures. Test under load, adjust based on metrics.

Read replica: Read-only copy of primary database, asynchronously replicated. Primary handles writes, replicas handle reads. Use cases: 1) Read-heavy applications (10:1 read:write ratio), 2) Analytics queries (don't impact production), 3) Geographic distribution (replica closer to users), 4) Disaster recovery backup. Setup: Primary → async replication → Replica(s). Application logic: Write requests → Primary, Read requests → Replicas (round-robin). Challenges: Replication lag (eventual consistency, replica may be few seconds behind), read-after-write inconsistency (write to primary, immediately read from replica may not see new data). Solution: Read from primary after write, or implement retry logic. Example: E-commerce - product catalog reads from replica, checkout writes to primary.

Locking prevents race conditions in concurrent updates. Pessimistic Locking: Lock record during read, preventing other transactions from modifying. Example: SELECT * FROM products WHERE id=1 FOR UPDATE; (row locked until transaction commits). High isolation but lower concurrency. Use when: High contention, conflicts likely. OptimContinue11:11 AMLocking: Don't lock, instead check if record changed before update. Add version column: UPDATE products SET quantity=quantity-1, version=version+1 WHERE id=1 AND version=5; If 0 rows affected, another transaction modified it, retry. Better concurrency but requires retry logic. Use when: Low contention, conflicts rare. Example: E-commerce inventory - optimistic (rare conflicts). Bank account - pessimistic (high contention).

Multi-tenancy: Single application instance serves multiple customers (tenants). Approaches: 1) Shared DB, Shared Schema: All tenants share tables, tenant_id column discriminates data. Pros: Cost-effective, easy maintenance. Cons: Security risk (data leakage), noisy neighbor, hard to scale per tenant. 2) Shared DB, Separate Schema: Each tenant has own schema (schema1.users, schema2.users). Balance of isolation and cost. 3) Separate DB per tenant: Complete isolation. Pros: Better security, scale independently. Cons: Higher cost, complex management. Implementation: Middleware sets tenant context: req.tenant = extractTenant(req); All queries include: WHERE tenant_id = req.tenant.id. Row-Level Security (RLS) in PostgreSQL enforces automatically. Choose based on: Security requirements, scale, budget.

Both split data across multiple storage locations. Partitioning: Split data within single database instance. Types: 1) Horizontal (range): orders_2024, orders_2025. 2) Vertical: Split columns (hot vs cold data). Benefits: Query performance (scan less data), maintenance (drop old partitions). Transparent to application. Sharding: Split data across multiple database servers. Each shard is independent DB. Example: Users 1-1M on Shard1, 1M-2M on Shard2. Application aware (routing logic needed). Benefits: Horizontal scaling, handle massive datasets. Cons: Complex queries across shards, rebalancing. Use partitioning for: Large tables, time-series data. Use sharding for: Scaling beyond single server, millions of users/records.

Backup strategies: 1) Full backup: Complete database snapshot (daily/weekly). Large storage, slow restore. 2) Incremental: Only changes since last backup (hourly). Fast, smaller, requires full + all incrementals to restore. 3) Differential: Changes since last full backup (daily). Faster restore than incremental. Point-in-Time Recovery (PITR): Restore to exact moment. Uses full backup + transaction logs. Example PostgreSQL: pg_basebackup (full) + WAL archiving (continuous). Disaster Recovery Plan: 1) RTO (Recovery Time Objective): How long can system be down? 2) RPO (Recovery Point Objective): How much data loss acceptable? 3) Automated backups with monitoring/alerts, 4) Regular restore testing (critical!), 5) Offsite/multi-region storage, 6) Documented procedures. Tools: AWS RDS snapshots, Barman, pgBackRest.

VACUUM reclaims storage from dead tuples in PostgreSQL. How PostgreSQL works: UPDATE/DELETE don't physically remove rows, they mark as dead. New version created for UPDATE. Dead tuples accumulate, waste space, slow queries. VACUUM: 1) Removes dead tuples, 2) Frees space for reuse (doesn't return to OS), 3) Updates statistics for query planner. VACUUM FULL: Returns space to OS but locks table. AUTOVACUUM: Runs automatically when threshold of dead tuples reached. Monitor: SELECT schemaname, relname, n_dead_tup FROM pg_stat_user_tables ORDER BY n_dead_tup DESC; If autovacuum can't keep up: 1) Tune autovacuum_vacuum_scale_factor, 2) Increase autovacuum workers, 3) Schedule manual VACUUM during off-peak. Critical for: High-write workloads.

EXPLAIN shows query execution plan (how DB will execute query). PostgreSQL: EXPLAIN ANALYZE SELECT * FROM users WHERE email='test@test.com'; Output: Seq Scan (full table scan) vs Index Scan (uses index), cost, actual time, rows. Optimization steps: 1) Identify slow queries (slow query log, APM), 2) Run EXPLAIN ANALYZE, 3) Look for: Seq Scan on large tables (add index), high cost, nested loop joins on large tables. 4) Create indexes: CREATE INDEX idx_email ON users(email); 5) Re-run EXPLAIN, verify Index Scan. Other techniques: Analyze query patterns, denormalize for read-heavy, partition large tables, update statistics (ANALYZE), avoid SELECT *, use LIMIT. Example: Seq Scan cost=1000 → Index Scan cost=10 (100x faster). Tools: pgAdmin, pg_stat_statements.

Hard Delete: Permanently remove record from database. DELETE FROM users WHERE id=1; Unrecoverable (unless backup). Soft Delete: Mark record as deleted without removing. Add deleted_at column (nullable timestamp). UPDATE users SET deleted_at=NOW() WHERE id=1; Queries: SELECT * FROM users WHERE deleted_at IS NULL; Pros: Data recovery, audit trail, maintain referential integrity, regulatory compliance. Cons: More complex queries, larger DB size, indexes affected. Implementation: 1) Add migration: ALTER TABLE users ADD deleted_at TIMESTAMP; 2) ORM support: Sequelize paranoid: true. 3) Views: CREATE VIEW active_users AS SELECT * FROM users WHERE deleted_at IS NULL; Use soft delete for: User data, financial records, audit requirements. Hard delete for: Personal data (GDPR right to deletion), logs, temporary data.

Isolation levels control concurrent transaction visibility. PostgreSQL/MySQL: 1) READ UNCOMMITTED: Can read uncommitted changes (dirty reads). Rarely used. 2) READ COMMITTED (PostgreSQL default): Sees only committed data. Prevents dirty reads. Each query sees latest committed data (non-repeatable reads possible). Use: Most applications. 3) REPEATABLE READ (MySQL default): Same query returns same results during transaction. Prevents non-repeatable reads. Phantom reads still possible (new rows inserted). Use: Financial calculations, reports. 4) SERIALIZABLE: Strongest. Transactions appear sequential. Prevents phantom reads. Use: Critical financial transactions. Trade-off: Stronger isolation = lower concurrency, higher chance of conflicts. Example: BEGIN TRANSACTION ISOLATION LEVEL SERIALIZABLE; SELECT SUM(balance) FROM accounts; /* Guaranteed consistent snapshot */ COMMIT; Choose based on: Consistency vs performance requirements.

Production-ready connection pooling: 1) Library: pg-pool (PostgreSQL), mysql2 (MySQL). 2) Configuration: const pool = new Pool({ host: 'localhost', database: 'mydb', max: 20, // Max connections min: 5, // Always maintain idleTimeoutMillis: 30000, // Close idle after 30s connectionTimeoutMillis: 5000, // Wait max 5s for connection maxUses: 7500, // Recycle connection after 7500 uses allowExitOnIdle: false }); 3) Error handling: pool.on('error', (err, client) => { logger.error('Unexpected pool error', err); }); 4) Graceful shutdown: process.on('SIGTERM', async () => { await pool.end(); }); 5) Monitoring: Track pool.totalCount, pool.idleCount, pool.waitingCount. Alert if waitingCount high (pool exhausted). 6) Testing: Simulate load, tune max based on DB server capacity. Formula: max = Tn × (Cm − 1) + 1 (Tn=num threads, Cm=num cores). Don't over-provision (wastes DB resources).

Database partitioning splits data for performance. Horizontal Partitioning (Sharding): Split rows across tables/servers. Example: orders_2024 (rows with year=2024), orders_2025 (year=2025). Or users_0 (id % 10 = 0), users_1 (id % 10 = 1). Benefits: Faster queries (scan less data), parallel processing, scale storage. Use: Time-series data, large user bases. Vertical Partitioning: Split columns into separate tables. Example: users table split into users_core (id, email, password) and users_profile (id, bio, avatar, preferences). Benefits: Frequently accessed columns together, cold data separate, reduce row size. Use: Wide tables, distinct access patterns. Can combine: Horizontally partition users by region, vertically split each partition (core vs profile columns).

Deadlock: Two transactions waiting for each other's locks, stuck forever. Example: Transaction A locks row 1, waits for row 2. Transaction B locks row 2, waits for row 1. DB detects deadlock, aborts one transaction. Detection: Logs show 'deadlock detected' error. Prevention: 1) Consistent lock order: Always lock resources in same order (by ID). If need rows 5 and 3, lock 3 first then 5. 2) Keep transactions short: Less time holding locks. 3) Use appropriate isolation level: Read Committed instead of Serializable if possible. 4) Avoid user interaction in transactions: Don't wait for user input. 5) Use optimistic locking: Version-based updates instead of locks. Handling: Implement retry logic with exponential backoff. Example: try { /* transaction */ } catch (err) { if(err.code === '40P01') { /* retry */ } }. Monitor deadlock frequency, investigate if increasing.

Foreign key constraint maintains referential integrity between tables. Example: orders table has user_id column referencing users.id. Ensures user_id always exists in users table. Creation: ALTER TABLE orders ADD CONSTRAINT fk_user FOREIGN KEY (user_id) REFERENCES users(id); Cascading actions: 1) ON DELETE CASCADE: Delete user → auto-delete their orders. 2) ON DELETE SET NULL: Delete user → set orders.user_id to NULL. 3) ON DELETE RESTRICT/NO ACTION: Prevent delete if orders exist. 4) ON UPDATE CASCADE: Update user.id → auto-update orders.user_id. Example: FOREIGN KEY (user_id) REFERENCES users(id) ON DELETE CASCADE ON UPDATE CASCADE. Use cases: CASCADE for dependent data (order items when order deleted), SET NULL for optional relations, RESTRICT for critical data (prevent accidental deletes). Trade-off: Enforce integrity vs flexibility. Some systems skip FKs for performance (enforce in application).

Full-text search finds documents matching text query (more than simple LIKE). PostgreSQL: 1) Create tsvector column: ALTER TABLE articles ADD COLUMN tsv tsvector; 2) Populate: UPDATE articles SET tsv = to_tsvector('english', title || ' ' || content); 3) Create index: CREATE INDEX tsv_idx ON articles USING GIN(tsv); 4) Query: SELECT * FROM articles WHERE tsv @@ to_tsquery('english', 'postgres & performance'); Features: Stemming (running → run), ranking, phrase search. MySQL: Full-text indexes on InnoDB. Advanced: ElasticSearch for complex search (typo tolerance, faceting, relevance scoring). Example: Product search across name, description, tags. Trade-offs: PostgreSQL full-text sufficient for simple use cases. ElasticSearch for advanced (autocomplete, fuzzy, aggregations) but adds infrastructure complexity. Monitor: Search performance, query patterns, relevance quality.

Prepared statement: Pre-compiled SQL with placeholders, executed with parameters. Example: const stmt = await db.prepare('SELECT * FROM users WHERE email = ?'); const user = await stmt.get('test@test.com'); Benefits: 1) SQL injection prevention: Parameters properly escaped, can't inject malicious SQL. 2) Performance: Query parsed once, executed multiple times (20-30% faster for repeated queries). 3) Type safety: Driver handles type conversion. Example vulnerability: const email = req.body.email; db.query(`SELECT * FROM users WHERE email = '${email}'`); // UNSAFE! email = "' OR '1'='1" bypasses auth. Prevention: const email = req.body.email; db.query('SELECT * FROM users WHERE email = ?', [email]); // Safe. Placeholders: ? (MySQL, SQLite), $1, $2 (PostgreSQL). ORMs use prepared statements automatically. Always use for user input!

Query plan cache: Database stores execution plan for queries to avoid re-planning. Process: 1) First execution: Parse SQL → Generate plan → Execute → Cache plan. 2) Subsequent executions: Retrieve cached plan → Execute (skips parse/planning). Benefits: Faster query execution (skip expensive planning step), reduced CPU usage. When it helps: Repeated queries with same structure (different parameters OK if using prepared statements). Example: SELECT * FROM users WHERE id = ? executed 1000 times, plan cached after first. When it doesn't help: Ad-hoc queries (different each time), complex queries where planning time small vs execution. Cache invalidation: Schema changes (ALTER TABLE), statistics updates (ANALYZE), cache size limits (LRU eviction). PostgreSQL: Plan cached per session for prepared statements. MySQL: Query cache (deprecated in 8.0, use prepared statements instead). Monitor: pg_stat_statements shows execution counts.

Authentication: Verifying WHO you are (identity verification). Example: Login with username/password, checking if credentials match database. Authorization: Verifying WHAT you can do (permission verification). Example: After login, checking if user has admin role to access /admin endpoint. Flow: Authentication happens first, then authorization. Analogy: Authentication is showing your ID at airport, authorization is checking if your ticket allows business class access. Both are needed for secure systems.

JWT is a compact, self-contained token for securely transmitting information between parties. Structure: header.payload.signature. Header: {alg: 'HS256', typ: 'JWT'} - algorithm and type. Payload: {userId: 123, role: 'admin', exp: 1234567890} - claims/data. Signature: HMACSHA256(base64(header) + '.' + base64(payload), secret) - verifies integrity. Flow: 1) User logs in, server generates JWT, 2) Client stores JWT (localStorage), 3) Client sends JWT in header: Authorization: Bearer <token>, 4) Server verifies signature and extracts payload. Stateless - no session storage needed.

OAuth 2.0 is an authorization framework for delegated access (login with Google/Facebook). Roles: Resource Owner (user), Client (app), Authorization Server (Google), Resource Server (user data). Flow: 1) User clicks 'Login with Google', 2) Redirected to Google login, 3) User authorizes app, 4) Google redirects back with authorization code, 5) App exchanges code for access token, 6) App uses token to fetch user data from Google API. Grant types: Authorization Code (most secure), Implicit (legacy), Client Credentials (machine-to-machine), Refresh Token (get new access token).

Session-based: Server stores session data, sends session ID to client in cookie. Stateful - server must maintain session store (Redis/DB). Scaling requires sticky sessions or shared session store. Token-based (JWT): Server generates signed token, client stores it. Stateless - server doesn't store anything, verifies signature each request. Better for microservices and mobile apps. Trade-off: Sessions allow server-side revocation (logout), JWTs can't be invalidated until expiry (unless using blacklist). Use sessions for traditional web apps, JWTs for APIs and mobile.

Never store plain text passwords! Steps: 1) Use strong hashing algorithm: bcrypt, argon2, scrypt (NOT MD5/SHA1). 2) Add salt (random string) to prevent rainbow table attacks. 3) Use high cost factor (bcrypt rounds). Example with bcrypt: const bcrypt = require('bcrypt'); const saltRounds = 10; const hashedPassword = await bcrypt.hash(plainPassword, saltRounds); // Store hashedPassword in DB. Login verification: const match = await bcrypt.compare(loginPassword, storedHash); if(match) { /* allow access */ }. Salt ensures same password has different hash for different users.

RBAC assigns permissions to roles, and roles to users. Example: Roles: admin, editor, viewer. Permissions: admin can CREATE/READ/UPDATE/DELETE, editor can READ/UPDATE, viewer can only READ. User gets role assignment: user123 → editor role. Implementation: Store user roles in DB, check role in middleware: function authorize(roles) { return (req, res, next) => { if(!roles.includes(req.user.role)) return res.status(403).json({error: 'Forbidden'}); next(); }}; app.delete('/posts/:id', authorize(['admin']), deletePost);. More scalable than assigning permissions directly to users.

Refresh tokens allow obtaining new access tokens without re-authentication. Problem: If access token has long expiry, security risk. If short expiry, user must login frequently. Solution: Access token (short expiry, 15 min) + Refresh token (long expiry, 7 days). Flow: 1) Login returns both tokens, 2) Use access token for requests, 3) When access token expires, send refresh token to /refresh endpoint, 4) Server validates refresh token, issues new access token, 5) If refresh token expires/invalid, user must login again. Store refresh token securely (httpOnly cookie), rotate on each use for better security.

Challenge: JWTs are stateless, can't be invalidated server-side. Solutions: 1) Client-side: Delete token from storage (localStorage/cookie). Simple but token still valid if leaked. 2) Token Blacklist: On logout, store token in Redis with expiry = token's remaining time. Check blacklist on each request. 3) Short expiry + Refresh tokens: Access token expires quickly, logout invalidates refresh token. 4) Token versioning: Store token version in DB, increment on logout, reject tokens with old version. Example: app.post('/logout', (req, res) => { const token = req.headers.authorization; redis.setex(token, tokenExpiry, 'blacklisted'); res.json({message: 'Logged out'}); });

2FA adds a second verification step beyond password for enhanced security. Types: 1) SMS OTP: Send code to phone, 2) Authenticator App (TOTP): Google Authenticator generates time-based codes, 3) Email verification, 4) Biometric. Implementation flow: 1) User enables 2FA, 2) Generate secret key, 3) User scans QR code with authenticator app, 4) On login, verify password then prompt for 6-digit code, 5) Verify code using library (speakeasy). Example: const speakeasy = require('speakeasy'); const verified = speakeasy.totp.verify({secret: userSecret, encoding: 'base32', token: userInputCode, window: 2});

SSO allows users to authenticate once and access multiple applications. Example: Google account gives access to Gmail, YouTube, Drive. How it works: 1) User accesses App1, 2) Redirected to central SSO server (Identity Provider), 3) User logs in once, 4) SSO server creates session and returns token, 5) User accesses App2, 6) App2 checks with SSO server, 7) SSO server confirms authentication, no re-login needed. Protocols: SAML (enterprise), OAuth 2.0 + OpenID Connect (modern). Benefits: Better UX, centralized user management, enhanced security (one strong auth point).

Caching stores frequently accessed data in fast storage (memory) to reduce expensive operations like database queries or API calls. Benefits: Reduced latency (microseconds vs milliseconds), lower database load, improved scalability, cost savings. Types: 1) Application cache (in-memory), 2) Database query cache, 3) CDN cache (static assets), 4) Browser cache. Example: Instead of querying DB for user profile on every request, cache it in Redis for 5 minutes. Cache hit: Return from cache (fast). Cache miss: Query DB, store in cache, return (slower first time, fast thereafter).

1) Cache-Aside (Lazy Loading): App checks cache first, if miss loads from DB and updates cache. Most common. 2) Write-Through: Write to cache and DB simultaneously. Data always consistent but slower writes. 3) Write-Back (Write-Behind): Write to cache, async write to DB later. Fast writes but risk of data loss. 4) Refresh-Ahead: Proactively refresh cache before expiry based on access patterns. Example: For read-heavy app (news feed), use cache-aside. For write-heavy app (analytics), use write-back. Choose based on read/write ratio and consistency requirements.

Redis is an in-memory data structure store used as database, cache, and message broker. Features: Supports strings, hashes, lists, sets, sorted sets. Persistence options (RDB snapshots, AOF logs). Atomic operations. Pub/Sub messaging. Use cases: 1) Caching: Session storage, API response cache, 2) Rate limiting: Track request counts, 3) Real-time analytics: Leaderboards, counters, 4) Queue: Background jobs, 5) Pub/Sub: Chat applications. Example: redis.set('user:123', JSON.stringify(userData), 'EX', 300); // Cache for 5 min

Cache invalidation removes or updates stale cached data. Hardest problem in computer science! Strategies: 1) TTL (Time To Live): Auto-expire after fixed time. Simple but may serve stale data. 2) Event-based: Invalidate cache when data changes. Example: When user updates profile, delete cached profile. 3) Tag-based: Group related cache entries, invalidate by tag. 4) Version-based: Append version to cache key (user:123:v2). Challenges: Race conditions, distributed system coordination. Example: redis.del('user:123'); // Invalidate after user update

Cache Stampede: When cached item expires, multiple requests simultaneously query DB (thundering herd). Solution: Use locks - first request fetches from DB while others wait, then all get cached data. Cache Penetration: Requests for non-existent data bypass cache and hit DB repeatedly (malicious queries). Solution: Cache null results with short TTL, use Bloom filter to check existence before querying DB. Example: const lock = await redis.set('lock:user:123', '1', 'NX', 'EX', 5); if(lock) { /* fetch from DB */ } else { /* wait and retry */ }

CDN (Content Delivery Network) is a distributed network of servers that cache and serve static content (images, CSS, JS, videos) from locations closest to users. Benefits: 1) Reduced latency (user in India gets content from India server vs US), 2) Lower bandwidth costs, 3) DDoS protection, 4) Offload traffic from origin server. Example: User in Mumbai requests image from yoursite.com → CDN serves from Mumbai edge server instead of origin in US (50ms vs 300ms). Popular CDNs: Cloudflare, AWS CloudFront, Akamai. Cache static assets, use cache headers: Cache-Control: max-age=31536000.

Query caching: Store query results in fast storage (Redis) to avoid repeated DB hits. Implementation: const cacheKey = `query:users:${filters}`; let result = await redis.get(cacheKey); if(!result) { result = await db.query('SELECT * FROM users WHERE ...'); await redis.setex(cacheKey, 300, JSON.stringify(result)); // Cache 5 min } else { result = JSON.parse(result); } Strategies: 1) Cache-aside: Check cache → if miss, query DB → update cache. 2) Time-based: TTL (300 seconds), refresh automatically. 3) Event-based invalidation: On user update, delete cached user queries. redis.del('query:users:*'); 4) Cache warming: Proactively populate cache (before expiry). Considerations: Cache key design (include all params), serialization (JSON stringify), cache size limits (LRU eviction), consistency (stale data acceptable?). Best for: Expensive queries, rarely changing data. Example: Product catalog, configuration data.

Message Queue is asynchronous communication pattern where producers send messages to a queue and consumers process them independently. Benefits: Decoupling (producer/consumer don't know each other), load leveling (handle traffic spikes), reliability (messages persist if consumer crashes), scalability. Use cases: 1) Email sending (don't block user request), 2) Image processing (resize uploaded images), 3) Order processing, 4) Logging. Example: User uploads video → Send message to queue → Background worker transcodes video. Technologies: RabbitMQ, Apache Kafka, AWS SQS, Redis Pub/Sub.

Message Queue (Point-to-Point): One producer, one consumer. Message delivered to exactly one consumer and removed from queue. Example: Order processing - each order processed by one worker. Pub/Sub (Publish-Subscribe): One producer, multiple subscribers. Message delivered to all subscribers. Example: User posts update → notify followers, send email, update analytics. Queue guarantees exactly-once delivery. Pub/Sub allows broadcast. Use Queue for task distribution, Pub/Sub for event notifications. Redis supports both: LPUSH/RPOP for queue, PUBLISH/SUBSCRIBE for pub/sub.

Kafka is a distributed event streaming platform for high-throughput, fault-tolerant message processing. Features: Persistent logs, horizontal scalability, real-time processing, replay capability. Architecture: Producers → Topics (partitions) → Consumer Groups. Use cases: 1) Activity tracking (user clicks), 2) Metrics collection, 3) Log aggregation, 4) Stream processing (fraud detection), 5) Event sourcing, 6) Microservices communication. Example: E-commerce: Order placed → Kafka topic → Multiple consumers (inventory service, email service, analytics). Difference from RabbitMQ: Kafka is log-based (retains messages), RabbitMQ is queue-based (deletes after consumption).

Idempotency means performing an operation multiple times has the same effect as performing it once. Critical for message queues due to at-least-once delivery (messages may be delivered multiple times). Example: Payment processing - charging user twice is bad! Solution: 1) Use unique request IDs - check if already processed before executing, 2) Database constraints (unique keys), 3) Atomic operations. Implementation: Before processing payment, check if payment_id exists in DB. If yes, skip. If no, process and insert payment_id. Example: if(await db.payments.findOne({id: msg.paymentId})) return; // Already processed

DLQ stores messages that failed processing after multiple retry attempts. Prevents poison messages (messages that crash consumer) from blocking the queue. Flow: 1) Message fails processing, 2) Retry 3 times, 3) After 3 failures, move to DLQ, 4) Alert developers to investigate. Benefits: Queue continues processing other messages, failed messages preserved for debugging. Example in AWS SQS: Main queue → maxReceiveCount: 3 → Move to DLQ. Periodically review DLQ to fix bugs causing failures. Example: Invalid JSON format, external API timeout, business logic error.

Microservices break applications into small, independent services. Characteristics: 1) Single Responsibility: Each service does one thing well, 2) Independent Deployment: Deploy without affecting others, 3) Decentralized Data: Each service owns its database, 4) Technology Agnostic: Different services can use different tech stacks, 5) Failure Isolation: One service failure doesn't crash entire system, 6) API Communication: Services communicate via REST/gRPC/messaging. Example: Netflix - recommendation service, streaming service, user service all independent. Challenges: Distributed system complexity, network latency, data consistency.

Communication patterns: 1) Synchronous (Request/Response): REST APIs, gRPC. Service A calls Service B directly, waits for response. Simple but creates coupling. Example: Order service calls Inventory service to check stock. 2) Asynchronous (Event-Driven): Message queues (RabbitMQ, Kafka). Service A publishes event, Service B subscribes. Loose coupling, better fault tolerance. Example: Order placed event → Email service, Inventory service, Analytics service all react independently. 3) Service Mesh: Dedicated infrastructure layer (Istio, Linkerd) handles service-to-service communication, load balancing, retries, circuit breaking.

API Gateway is a single entry point for all client requests to microservices. Responsibilities: 1) Routing: Forward requests to appropriate service, 2) Authentication/Authorization: Centralized security, 3) Rate Limiting: Prevent abuse, 4) Load Balancing: Distribute traffic, 5) Request/Response Transformation: Aggregate multiple service calls, 6) Caching: Cache responses, 7) Logging/Monitoring: Track requests. Example: Mobile app → API Gateway → User Service/Order Service/Payment Service. Benefits: Simplified client, reduced round trips (gateway can aggregate data from multiple services). Technologies: Kong, AWS API Gateway, Express Gateway, Nginx.

Service Discovery allows services to find and communicate with each other dynamically. Problem: In microservices, service instances (IP addresses) change frequently due to scaling, failures, deployments. Hard-coding IPs doesn't work. Solution: Services register with Service Registry (Consul, Eureka, etcd), other services query registry to find instances. Patterns: 1) Client-Side Discovery: Client queries registry, chooses instance. 2) Server-Side Discovery: Load balancer queries registry. Example: Order service needs Payment service → Queries Consul → Gets http://payment:3001 → Makes request. Enables dynamic scaling and fault tolerance.

Saga pattern manages transactions across multiple microservices without distributed locks. Problem: Can't use traditional ACID transactions across services (different databases). Solution: Break transaction into sequence of local transactions, each with compensating transaction for rollback. Types: 1) Choreography: Each service produces/listens to events. Order created → Payment processed → Inventory reserved. If payment fails → Rollback order. 2) Orchestration: Central coordinator manages saga. Example: Order Saga orchestrator calls Payment → Inventory → Shipping. If any fails, executes compensating transactions in reverse. Technologies: Temporal, Camunda.

Circuit Breaker prevents cascading failures in distributed systems. States: 1) Closed (normal): Requests pass through. If failures exceed threshold, opens circuit. 2) Open: Immediately fail requests without calling service (fail fast). After timeout, enters half-open. 3) Half-Open: Try a few requests. If succeed, close circuit. If fail, stay open. Example: Payment service down → After 5 failures, circuit opens → User gets 'service unavailable' immediately instead of waiting for timeout → After 30 sec, try again. Benefits: Faster failures, service recovery time, prevent resource exhaustion. Libraries: Netflix Hystrix, resilience4j, Polly.

Correlation ID tracks request across services in distributed systems. Implementation: 1) Generate unique ID (UUID) at entry point (API Gateway), 2) Propagate ID through all services via headers (X-Correlation-ID), 3) Include in all logs, 4) Return in response headers for debugging. Example middleware: const {v4: uuidv4} = require('uuid'); app.use((req, res, next) => { req.correlationId = req.headers['x-correlation-id'] || uuidv4(); res.setHeader('X-Correlation-ID', req.correlationId); next(); }); Logger: logger.info({correlationId: req.correlationId, message: 'Processing request'}); When calling other services: axios.get('...', {headers: {'X-Correlation-ID': req.correlationId}}); Benefits: End-to-end tracing, easier debugging, request timeline visualization.

Outbox Pattern ensures reliable event publishing in distributed systems. Problem: Update DB and publish event - if event publish fails, DB inconsistent with event stream. Solution: 1) Store events in outbox table in same transaction as business data, 2) Separate process reads outbox, publishes to message broker, 3) Mark events as published. Example: BEGIN TRANSACTION; INSERT INTO orders (id, userId, total) VALUES (1, 123, 100); INSERT INTO outbox (id, eventType, payload) VALUES (uuid(), 'OrderCreated', {...}); COMMIT; Worker: SELECT * FROM outbox WHERE published = false; publish to Kafka; UPDATE outbox SET published = true WHERE id = ...; Benefits: Exactly-once delivery guarantee (at-least-once with deduplication), maintains consistency. Use: Event-driven architectures, CQRS, saga pattern.

API Gateway: Single entry point for clients, routes to appropriate microservices. Already answered in Q49, but additional details: Responsibilities beyond routing: 1) Request aggregation: Combine responses from multiple services (mobile app needs user + orders + recommendations → gateway makes 3 calls, returns combined response), 2) Protocol translation: WebSocket to HTTP, gRPC to REST, 3) Response transformation: Format data for different clients (mobile vs web), 4) API composition: Backend for Frontend (BFF) pattern - separate gateway per client type. Implementation: Custom (Express/Fastify), Managed (AWS API Gateway, Kong, Apigee). Example: gateway.get('/user-dashboard/:id', async (req, res) => { const [user, orders, recommendations] = await Promise.all([userService.get(req.params.id), orderService.getByUser(req.params.id), recommendationService.get(req.params.id)]); res.json({user, orders, recommendations}); });

Bulkhead Pattern: Isolate resources to prevent cascading failures (like bulkheads in ships prevent entire ship from sinking). Example: Application has 100 threads. Without bulkhead: PaymentService slow → All 100 threads stuck waiting for payment → Entire app unresponsive. With bulkhead: Allocate 20 threads for payments, 20 for inventory, 60 for other. PaymentService slow → Only 20 threads affected, other services still work. Implementation: 1) Thread pools: Separate thread pool per service dependency. 2) Connection pools: Separate DB connection pool per tenant. 3) Circuit breakers: Per-service circuit breakers (already isolates). Example with p-limit: const paymentLimit = pLimit(5); // Max 5 concurrent const inventoryLimit = pLimit(10); paymentLimit(() => callPaymentService()); Benefits: Fault isolation, predictable resource allocation, graceful degradation. Use: High-traffic systems, multiple service dependencies.

Sidecar Pattern: Deploy helper container alongside main application container (like sidecar on motorcycle). Sidecar handles cross-cutting concerns: logging, monitoring, service mesh, config management. Example: Main container (Node.js app) + Sidecar container (Envoy proxy for traffic management). Benefits: 1) Separation of concerns: App focuses on business logic, sidecar on infrastructure, 2) Language-agnostic: Same sidecar works with any language, 3) Reusability: Same sidecar across services. Use cases: 1) Service mesh (Istio, Linkerd): Sidecar proxies handle service-to-service communication, mTLS, circuit breaking, 2) Log aggregation: Sidecar collects logs, ships to centralized system, 3) Configuration: Sidecar pulls config, provides to app. Kubernetes: Define sidecar in same Pod. Example: Pod with app container + Envoy sidecar. Trade-off: Resource overhead (extra containers), complexity.

Ambassador Pattern: Proxy container handles outgoing requests from main application (outbound proxy). Like Sidecar but specifically for outbound traffic. Use cases: 1) Service discovery: App sends requests to ambassador, ambassador resolves service location. 2) Retry logic: Ambassador implements exponential backoff. 3) Circuit breaking: Ambassador monitors service health, fails fast. 4) Monitoring: Ambassador logs outbound requests. 5) Protocol translation: App speaks HTTP, ambassador translates to gRPC. Example: App → Ambassador (localhost:8080) → Ambassador resolves → External Service. Benefits: Offload infrastructure concerns, language-agnostic, reusability across services. Implementation: Sidecar container (Envoy, Nginx), app configured to route through localhost proxy. Kubernetes: Ambassador container in same Pod. Difference from Sidecar: Ambassador specifically for outbound, Sidecar general-purpose.

Vertical Scaling (Scale Up): Add more power to existing machine (CPU, RAM, Disk). Pros: Simple, no code changes. Cons: Hardware limits, single point of failure, expensive. Example: Upgrade server from 8GB to 32GB RAM. Horizontal Scaling (Scale Out): Add more machines to distribute load. Pros: Unlimited scaling, fault tolerance, cost-effective (commodity hardware). Cons: Complexity (load balancing, distributed data). Example: Add more servers behind load balancer. Modern apps prefer horizontal scaling. Stateless services scale easily. Databases are harder (sharding/replication needed).

Load Balancer distributes incoming requests across multiple servers. Benefits: High availability, improved performance, scalability. Algorithms: 1) Round Robin: Sequential distribution (server1, server2, server3, repeat). Simple but doesn't consider server load. 2) Least Connections: Route to server with fewest active connections. Good for long-lived connections. 3) IP Hash: Hash client IP, route to same server (session affinity). 4) Weighted Round Robin: Servers get weight based on capacity. 5) Least Response Time: Route to fastest server. Types: Layer 4 (TCP/UDP), Layer 7 (HTTP). Technologies: Nginx, HAProxy, AWS ELB, Google Cloud Load Balancer.

CAP Theorem: In a distributed system, you can have at most 2 of 3: Consistency (all nodes see same data simultaneously), Availability (every request gets response, even if some nodes are down), Partition Tolerance (system works despite network failures). Since network failures happen in distributed systems, P is mandatory, so choose between C and A. CP Systems: Wait for all nodes to sync before responding. Consistent but may be unavailable during partition. Example: MongoDB, HBase. AP Systems: Respond immediately from available nodes. Available but may serve stale data (eventual consistency). Example: Cassandra, DynamoDB. Choose based on use case: Banking (CP), Social media (AP).

Requirements: Shorten long URL, redirect to original URL, analytics. Approach: 1) Generate unique short code (6-7 characters = 62^7 = 3.5 trillion URLs). Hash long URL using MD5/SHA256, take first 7 chars (collision possible). Or use base62 encoding of auto-increment ID (no collision). 2) Store mapping: {shortCode: 'abc123', longURL: 'https://...', clicks: 0, createdAt: ...} in database. Use NoSQL (DynamoDB/MongoDB) for fast lookups. 3) Redirect: GET /:shortCode → lookup DB → 301 redirect to longURL. 4) Scale: Cache popular URLs in Redis, use CDN, shard DB by shortCode. 5) Analytics: Increment click counter, store user-agent, location. Handle: Custom URLs, expiration, rate limiting.

Requirements: Limit requests per user (100 req/min). Algorithms: 1) Token Bucket: Bucket has tokens, request consumes token. Tokens refill at fixed rate. Allows bursts. 2) Sliding Window: Count requests in rolling time window. More accurate than fixed window. Implementation with Redis: Use sorted set with timestamps. Example: ZADD user:123 currentTimestamp requestId; ZREMRANGEBYSCORE user:123 0 (currentTime - 60sec); count = ZCARD user:123; if(count > 100) reject; else allow. Distributed: Use Redis (shared state). Handle: Different limits per tier (free/premium), IP-based limiting, exponential backoff.

Consistent Hashing minimizes data movement when nodes are added/removed. Traditional hash: hash(key) % N (where N = number of servers). Problem: Adding/removing server rehashes almost all keys. Consistent Hashing: Hash both keys and servers onto a ring (0 to 2^32-1). Key goes to next server clockwise on ring. Benefits: Adding/removing server only affects 1/N keys (minimal redistribution). Virtual nodes: Each physical server has multiple positions on ring for better distribution. Use cases: Distributed caching (Memcached), load balancing, data sharding. Example: 3 servers, add 4th server → Only ~25% keys move (vs 75% with traditional hashing).

Strangler Fig Pattern: Gradually replace legacy system with new system. Named after strangler fig tree that grows around another tree. Steps: 1) Create facade/proxy in front of legacy system, 2) Implement new features in new system, 3) Gradually migrate existing features (one route at a time), 4) Proxy routes requests to old or new system, 5) Eventually remove old system. Example: Legacy monolith → API Gateway routes /users to new service, /orders still to monolith → Migrate /orders → Decommission monolith. Benefits: Lower risk (incremental), continuous delivery, no big-bang rewrite, test in production. Challenges: Maintain two systems, data synchronization. Alternative to risky full rewrites. Used by: Soundcloud, GitHub for monolith→microservices migration.

Anti-Corruption Layer (ACL): Isolates your domain model from external systems/legacy code to prevent corruption. Problem: External system has different models, bad design, or frequently changes. Direct integration pollutes your clean domain. Solution: Create translation layer between your system and external system. ACL translates data formats, models, protocols. Example: Modern microservice integrates with legacy SOAP API. ACL: 1) Accepts domain objects (clean models), 2) Translates to SOAP format, 3) Calls legacy system, 4) Translates response back to domain objects, 5) Returns to application. Benefits: Protects domain model, isolates changes (external system changes, only ACL needs update), allows gradual migration. Implementation: Adapter/Facade pattern. Example: class LegacySystemAdapter { async getUser(userId) { const soapResponse = await legacyAPI.getUserSOAP(userId); return this.translateToDomain(soapResponse); } } Use when: Integrating with legacy systems, third-party APIs with bad models, bounded context integration (DDD).

SQL Injection: Attacker injects malicious SQL code through user input. Example: username = ' OR '1'='1' -- turns SELECT * FROM users WHERE name='$username' into SELECT * FROM users WHERE name='' OR '1'='1' --, returning all users. Prevention: 1) Parameterized Queries (Prepared Statements): const query = 'SELECT * FROM users WHERE name = ?'; db.query(query, [username]); 2) Use ORM (Sequelize, TypeORM) - they sanitize inputs. 3) Input validation - whitelist allowed characters. 4) Least privilege - DB user should have minimal permissions. 5) WAF (Web Application Firewall). Never concatenate user input into SQL strings!

XSS: Attacker injects malicious JavaScript into web pages viewed by other users. Types: 1) Stored XSS: Malicious script saved in DB (comment field). 2) Reflected XSS: Script in URL parameter, reflected in page. 3) DOM-based XSS: Client-side script manipulation. Example: Comment = <script>steal_cookies()</script>. Prevention: 1) Escape output - convert < to &lt;, > to &gt; before rendering. 2) Content Security Policy (CSP) header - blocks inline scripts. 3) HttpOnly cookies - JavaScript can't access. 4) Input validation - sanitize HTML. 5) Use frameworks that auto-escape (React, Angular). Never use innerHTML with user input, use textContent instead.

CSRF: Attacker tricks user into executing unwanted actions on authenticated site. Example: User logged into bank.com. Visits malicious site with <img src='bank.com/transfer?to=attacker&amount=1000'> - browser sends cookies automatically, executes transfer! Prevention: 1) CSRF Tokens: Server generates random token, embeds in form. Validate token on submission. 2) SameSite Cookie attribute: Cookie sent only for same-site requests (SameSite=Strict/Lax). 3) Check Referer/Origin headers. 4) Re-authentication for sensitive actions. Example: <form><input type='hidden' name='csrf_token' value='random123'> Server validates token matches session.

Security headers protect against common attacks. Key headers: 1) Content-Security-Policy: Prevents XSS by controlling resource loading. Example: CSP: default-src 'self' - only load from same origin. 2) X-Frame-Options: DENY/SAMEORIGIN - prevents clickjacking. 3) Strict-Transport-Security (HSTS): Forces HTTPS. max-age=31536000. 4) X-Content-Type-Options: nosniff - prevents MIME type sniffing. 5) Referrer-Policy: Controls referrer info leakage. Implementation in Express: helmet middleware app.use(helmet()). Test with securityheaders.com. These headers add defense-in-depth layers.

Least Privilege: Grant minimum permissions required for a task. Examples: 1) Database: App user has SELECT/INSERT on specific tables, not full admin. Read-only replica for analytics. 2) API: Mobile app can only access own user data, admin panel accesses all. 3) File system: Web server reads files but can't execute. 4) Cloud: IAM role for Lambda has only S3 read permission, not write/delete. Benefits: Reduced blast radius (if compromised, attacker limited), easier auditing, compliance. Implementation: Use role-based access, principle of zero trust, regularly review permissions. Example: AWS IAM policies with specific actions/resources, not *.

Never log sensitive data (passwords, credit cards, SSN, tokens). Risks: Logs stored long-term, accessible to many, may be sent to third-party services. Solutions: 1) Filter/Redact: Replace sensitive fields with [REDACTED]. Example: {email: 'user@example.com', password: '[REDACTED]'}. 2) Use structured logging (JSON) with automatic filtering. 3) Hash before logging: log.info({userId: hash(userId)}). 4) Encryption: Encrypt logs at rest. 5) Access controls on logs. 6) Audit logs separately (immutable, high security). Example Winston middleware: logger.add(new transports.File({filename: 'app.log', format: redactFormat()}));. Regularly audit logs for leaks.

Limiting request body size prevents DoS attacks (sending huge payloads), memory exhaustion. Implementation: 1) Express: app.use(express.json({limit: '10mb'})); // JSON bodies max 10MB app.use(express.urlencoded({limit: '10mb', extended: true})); 2) Raw body: app.use(express.raw({limit: '50mb', type: 'application/octet-stream'})); 3) File uploads (Multer): const upload = multer({limits: {fileSize: 5 * 1024 * 1024}}); // 5MB 4) Server level (Nginx): client_max_body_size 10M; Error handling: app.use((error, req, res, next) => { if(error.type === 'entity.too.large') { return res.status(413).json({error: 'Payload too large', maxSize: '10MB'}); } next(error); }); Set limits based on use case: APIs (1-10MB), file uploads (50-100MB), stream uploads (unlimited). Return 413 Payload Too Large with informative message. Monitor: Request sizes, alert on frequent 413s (attack or legitimate usage increase).

Row-Level Security (RLS) restricts which rows users can access based on their identity. PostgreSQL RLS: 1) Enable RLS: ALTER TABLE users ENABLE ROW LEVEL SECURITY; 2) Create policy: CREATE POLICY user_isolation ON users FOR ALL TO app_user USING (id = current_user_id()); -- current_user_id() is custom function returning user from session. 3) Grant access: GRANT ALL ON users TO app_user; Now app_user can only see/modify their own rows. Policies can be: FOR SELECT (read), FOR INSERT/UPDATE/DELETE (write), FOR ALL. Use cases: Multi-tenant SaaS (tenant isolation), user-specific data, healthcare (patient privacy). Benefits: Enforced at DB level (can't bypass in app code), centralized security, simplifies application logic. Implementation: Set session variable with user ID after auth, policies use session variable. Example function: CREATE FUNCTION current_user_id() RETURNS INTEGER AS $$ SELECT current_setting('app.user_id')::INTEGER; $$ LANGUAGE SQL; Set in app: await db.query("SET app.user_id = $1", [req.user.id]);

1) Indexing: Create indexes on frequently queried columns. Analyze query with EXPLAIN. 2) Avoid SELECT *: Fetch only needed columns. 3) Limit results: Use LIMIT/OFFSET for pagination. 4) Optimize JOINs: Use INNER JOIN over subqueries when possible, join on indexed columns. 5) Avoid N+1: Use eager loading. 6) Denormalization: For read-heavy apps, duplicate data to reduce joins. 7) Query caching: Cache frequent queries in Redis. 8) Partition large tables: Split by date/range. 9) Use EXPLAIN ANALYZE to identify slow parts. Example: Instead of SELECT * FROM orders WHERE user_id IN (SELECT id FROM users WHERE country='US'), use JOIN: SELECT o.* FROM orders o JOIN users u ON o.user_id=u.id WHERE u.country='US'.

Opening database connection is expensive (TCP handshake, authentication). Connection pooling maintains reusable connections. Benefits: Reduced latency (reuse instead of create), controlled resource usage (limit max connections), better throughput. Configuration: min (always maintained), max (upper limit), idle timeout (close unused), queue timeout (wait for available connection). Example in Node.js: const pool = new Pool({min: 5, max: 20, idleTimeoutMillis: 30000}); pool.query('SELECT * FROM users'); Without pooling: Each request creates new connection (slow). With pooling: Reuse from pool (fast). Monitor: Active connections, queue size. Tune based on load.

Memory leak: Memory allocated but never released, causing gradual memory increase until crash. Common causes in Node.js: 1) Global variables accumulating data, 2) Event listeners not removed, 3) Closures holding references, 4) Timers/intervals not cleared, 5) Large caches without eviction. Detection: Monitor memory usage (process.memoryUsage()), use heap snapshots, profilers (Chrome DevTools, clinic.js). Example leak: events.on('data', (data) => {cache.push(data)}); // cache grows forever. Fix: Set limits, remove listeners: events.off('data', handler), clear intervals: clearInterval(timer), use WeakMap for caches. Prevention: Code reviews, memory profiling in staging.

Event Loop allows Node.js to handle concurrent operations despite single-threaded JavaScript. Phases: 1) Timers: Execute setTimeout/setInterval callbacks, 2) Pending callbacks: System operations (TCP errors), 3) Poll: Retrieve new I/O events, execute I/O callbacks, 4) Check: setImmediate callbacks, 5) Close callbacks: socket.on('close'). Process: JavaScript → Call Stack → Web APIs (async ops) → Callback Queue → Event Loop moves callbacks to stack when empty. Example: fs.readFile() doesn't block - operation delegated to libuv thread pool, callback queued when done. process.nextTick() and Promise microtasks execute before next phase. Understanding this is crucial for performance.

Lazy loading defers loading of resources until needed. Backend examples: 1) Database relationships: Load related data only when accessed. In Sequelize: User.findOne({include: {model: Post, required: false}}) - loads posts only if accessed. 2) Pagination: Load data in chunks instead of all at once. 3) Image processing: Generate thumbnails on-demand, not at upload. 4) Module loading: Dynamic imports in Node.js - const module = await import('./heavy-module') only when needed. Benefits: Reduced initial load time, lower memory usage, better performance. Trade-off: Additional requests when accessed. Use for: Infrequently accessed data, large datasets, resource-intensive operations.

Background jobs handle time-consuming tasks asynchronously without blocking requests. Implementation: 1) Choose queue (BullMQ, Bee-Queue based on Redis), 2) Create job producer (API endpoint adds job to queue), 3) Create job consumer (worker processes jobs). Example with BullMQ: const queue = new Queue('email'); app.post('/signup', async (req, res) => { await User.create(req.body); await queue.add('welcome', {userId: user.id}); res.json({success: true}); }); Worker: queue.process('welcome', async (job) => { await sendEmail(job.data.userId); }); Features: Retries, concurrency, scheduling, prioritization. Use cases: Email sending, report generation, video processing, data imports. Benefits: Better user experience, horizontal scaling of workers.

Response compression reduces payload size, faster transfer. Implementation with compression middleware: const compression = require('compression'); app.use(compression({ filter: (req, res) => { if(req.headers['x-no-compression']) return false; return compression.filter(req, res); }, level: 6 // Compression level 0-9 (6 is good balance) })); How it works: 1) Client sends Accept-Encoding: gzip, deflate, br, 2) Server compresses response, 3) Server sends Content-Encoding: gzip, 4) Client decompresses. Algorithms: gzip (widely supported), brotli (better compression, newer), deflate. Benefits: Faster page loads (smaller payload), reduced bandwidth costs. Trade-offs: CPU overhead (compression), not worth for tiny responses. Best practices: 1) Compress text (JSON, HTML, CSS, JS), not images/videos (already compressed), 2) Threshold: Only compress responses > 1KB, 3) Tune compression level (9 = max compression, slow; 1 = min compression, fast). Typical savings: 60-80% for JSON.

HTTP caching: Use headers to control client/CDN caching. Headers: 1) Cache-Control: max-age=3600, public (cache for 1 hour, shared caches OK), max-age=0, private (don't cache), no-store (never cache), must-revalidate (revalidate when stale). 2) ETag: Hash of response content. Client sends If-None-Match: <etag> on subsequent requests. If unchanged, server returns 304 Not Modified (no body). 3) Last-Modified / If-Modified-Since: Similar to ETag, timestamp-based. Implementation: app.get('/api/products', (req, res) => { const etag = generateETag(products); if(req.headers['if-none-match'] === etag) return res.sendStatus(304); res.setHeader('Cache-Control', 'public, max-age=3600'); res.setHeader('ETag', etag); res.json(products); }); Benefits: Reduce bandwidth, faster responses, lower server load. Use: Static content (max-age=31536000), dynamic (short max-age, ETag for validation).

1) Use nouns, not verbs: /users (not /getUsers). HTTP method indicates action. 2) Plural names: /users not /user. 3) Hierarchical structure: /users/123/posts/456 for nested resources. 4) Use kebab-case: /user-profiles. 5) Versioning: /api/v1/users. 6) Filtering: /users?status=active. 7) Sorting: /users?sort=-createdAt (- for descending). 8) Pagination: /users?page=2&limit=20. 9) Field selection: /users?fields=name,email. 10) Avoid deep nesting: Max 2 levels. Bad: /users/123/posts/456/comments/789, Better: /comments/789. 11) Use standard status codes. 12) Consistent error format: {error: 'message', code: 'USER_NOT_FOUND'}.

1) Use appropriate HTTP status codes: 400 (validation), 401 (auth), 403 (forbidden), 404 (not found), 500 (server error). 2) Consistent error format: {error: {message: 'User not found', code: 'USER_NOT_FOUND', field: 'userId'}}. 3) Detailed messages in dev, generic in production (don't expose internals). 4) Log errors with context (request ID, user, timestamp). 5) Global error handler: app.use((err, req, res, next) => { logger.error(err); res.status(err.status || 500).json({error: err.message}); }). 6) Validation errors: List all fields: {errors: [{field: 'email', message: 'Invalid format'}]}. 7) Operational vs programmer errors: Handle operational (network), crash on programmer (bugs). 8) Error monitoring (Sentry).

API documentation explains endpoints, parameters, responses, authentication for developers. Importance: Easier adoption, fewer support requests, better collaboration. Tools: 1) Swagger/OpenAPI: YAML/JSON spec, auto-generates interactive docs. Example: Swagger UI at /api-docs. 2) Postman: Collections with examples, can generate docs. 3) Redoc: Clean OpenAPI renderer. 4) API Blueprint, RAML. 5) Code annotations: JSDoc, TypeScript decorators. Best practices: Include examples, error scenarios, rate limits, authentication steps, SDKs. Example OpenAPI: paths: /users/{id}: get: summary: 'Get user', parameters: [{name: id, in: path}], responses: {200: {description: 'Success'}}. Keep docs in sync with code (automation).

Request validation ensures incoming data meets expected format before processing. Prevents bugs, security issues, and database errors. Implementation with Joi (Node.js): const schema = Joi.object({email: Joi.string().email().required(), age: Joi.number().min(18).max(100)}); const {error, value} = schema.validate(req.body); if(error) return res.status(400).json({error: error.details}); Other tools: express-validator, Yup, class-validator (TypeScript). Validate: 1) Required fields, 2) Data types, 3) Format (email, URL), 4) Length/range, 5) Allowed values (enums). Example: POST /users {email: 'invalid', age: 15} → 400 Bad Request {errors: [{field: 'email', message: 'Invalid email'}, {field: 'age', message: 'Must be 18+'}]}.

Content negotiation allows client and server to agree on response format. Client specifies preferred format via headers: Accept: application/json or Accept: application/xml. Server responds with matching Content-Type. Implementation: app.get('/users/:id', (req, res) => { const user = getUser(req.params.id); if(req.accepts('json')) res.json(user); else if(req.accepts('xml')) res.type('xml').send(toXML(user)); else res.status(406).send('Not Acceptable'); }); Also applies to encoding (Accept-Encoding: gzip), language (Accept-Language: en-US). Benefits: Flexibility for different clients (web, mobile, IoT), backward compatibility. Best practice: Support JSON as default, add others if needed.

Implementation with Multer (Express): const multer = require('multer'); const storage = multer.diskStorage({ destination: './uploads/', filename: (req, file, cb) => cb(null, Date.now() + '-' + file.originalname) }); const upload = multer({ storage, limits: {fileSize: 5*1024*1024}, fileFilter: (req, file, cb) => { if(!file.mimetype.startsWith('image/')) return cb(new Error('Only images')); cb(null, true); } }); app.post('/upload', upload.single('avatar'), (req, res) => { res.json({path: req.file.path}); }); Best practices: 1) Validate file type, size, 2) Use cloud storage (S3, Cloudinary), 3) Generate unique filenames, 4) Virus scanning, 5) Serve via CDN, 6) Clean up old files. For large files: Use streaming, chunked uploads, presigned URLs (direct to S3).

API throttling limits request rate per user/IP to prevent abuse and ensure fair usage. Difference from rate limiting: Throttling slows down requests (delays), rate limiting blocks after limit. Implementation: 1) Token bucket: Users have bucket of tokens, request consumes token, tokens refill at fixed rate. When empty, delay request. 2) Leaky bucket: Requests queued, processed at fixed rate. Example with middleware: const delays = new Map(); app.use((req, res, next) => { const key = req.ip; const lastRequest = delays.get(key) || 0; const now = Date.now(); const timeSince = now - lastRequest; if(timeSince < 100) setTimeout(next, 100 - timeSince); else next(); delays.set(key, now); }); Use cases: Prevent scraping, protect against DDoS, manage load.

Cursor-based pagination: Use unique identifier (cursor) instead of offset. Offset problems: 1) Slow for large offsets (OFFSET 1000000), 2) Inconsistent results if data changes (new item inserted, page 2 shows page 1 item). Cursor solution: Return cursor with results, use for next page. Implementation: Query: SELECT * FROM posts WHERE created_at < :cursor ORDER BY created_at DESC LIMIT 20; Response: {data: [...], nextCursor: '2025-01-20T10:00:00Z'}. Next request: /posts?cursor=2025-01-20T10:00:00Z. If cursor is null, no more results. Encode cursor: Base64(JSON.stringify({created_at, id})) to handle tie-breakers. Benefits: Consistent results, better performance, handles real-time data. Cons: Can't jump to specific page. Use for: Infinite scroll, activity feeds, real-time data.

Idempotent operation: Multiple identical requests have same effect as single request. HTTP methods: GET, PUT, DELETE are naturally idempotent. POST is not (creating same resource twice creates duplicates). Implementation for POST: 1) Client generates idempotency key (UUID), sends in header: Idempotency-Key: abc-123. 2) Server checks if key exists in cache/DB. If yes, return previous result. If no, process request, store key with result, return. 3) TTL: Store key for 24 hours. Example: app.post('/payments', async (req, res) => { const key = req.headers['idempotency-key']; const cached = await redis.get(key); if(cached) return res.json(JSON.parse(cached)); const result = await processPayment(req.body); await redis.setex(key, 86400, JSON.stringify(result)); res.json(result); }); Critical for: Payments, order creation, any operation where duplicates cause issues.

Request deduplication prevents processing duplicate requests (e.g., user double-clicks submit). Similar to idempotency but focuses on concurrent duplicates. Implementation: 1) Generate request fingerprint: hash(userId + operation + data), 2) Store in cache with short TTL (few seconds), 3) If fingerprint exists, return in-progress or previous result. Example: const fingerprint = crypto.createHash('md5').update(req.user.id + req.url + JSON.stringify(req.body)).digest('hex'); const existing = await redis.get(fingerprint); if(existing) return res.status(409).json({error: 'Duplicate request'}); await redis.setex(fingerprint, 5, 'processing'); const result = await processRequest(req.body); await redis.del(fingerprint); res.json(result); For async operations: Return 'processing' status, allow client to poll. Use cases: Form submissions, payment processing, API rate limiting.

Rate limiting layers: 1) CDN/Edge: Cloudflare rate limiting (closest to user, protects infrastructure). 2) Load Balancer: Nginx limit_req_zone (before hitting app servers). 3) API Gateway: Kong, AWS API Gateway (centralized, applies to all services). 4) Application: Express middleware (granular control per endpoint). 5) Database: Query throttling (protect DB from expensive queries). Implementation strategy: Use multiple layers - CDN for DDoS, API Gateway for API limits, App for business logic (e.g., free tier 100 req/hour, premium 1000). Store state in Redis (distributed). Example Nginx: limit_req_zone $binary_remote_addr zone=mylimit:10m rate=10r/s; Benefits: Defense in depth, different limits per layer, better protection.

gRPC: High-performance RPC framework using HTTP/2 and Protocol Buffers (binary serialization). Features: 1) Faster than REST (binary vs JSON/text), 2) Strongly typed contracts (.proto files), 3) Bi-directional streaming, 4) Code generation for multiple languages. When to use gRPC: 1) Microservices internal communication (low latency critical), 2) Real-time streaming (video, gaming), 3) Mobile clients (bandwidth constrained), 4) Polyglot environments (multiple languages). When to use REST: 1) Public APIs (browser support, easier debugging), 2) Simple CRUD operations, 3) Human-readable data needed. Example: Google uses gRPC internally. Trade-off: gRPC harder to debug (binary), less tooling, no browser support. REST more universal.

Both schedule callback execution, but timing differs. process.nextTick(): Executes callback immediately after current operation, BEFORE Event Loop continues. Microtask queue - higher priority. setImmediate(): Executes callback in next iteration of Event Loop, in 'check' phase. Macrotask queue. Example: fs.readFile('file.txt', () => { console.log('1'); process.nextTick(() => console.log('2')); setImmediate(() => console.log('3')); }); Output: 1, 2, 3. Use nextTick() when you need to execute before I/O events, setImmediate() for deferring to next cycle. Warning: Too many nextTick() can starve Event Loop (I/O never executes). Prefer setImmediate() for most cases.

Streams process data in chunks instead of loading entire data in memory. Types: 1) Readable: Read data (fs.createReadStream, HTTP request), 2) Writable: Write data (fs.createWriteStream, HTTP response), 3) Duplex: Both read/write (TCP socket), 4) Transform: Modify data while reading/writing (compression). Example: const readable = fs.createReadStream('large.txt'); readable.on('data', chunk => console.log(chunk)); Benefits: Memory efficient (process 1GB file without loading all), faster time-to-first-byte. Use cases: File processing, video streaming, HTTP requests/responses. Pipe: readable.pipe(writable) connects streams. Example: fs.createReadStream('input.txt').pipe(fs.createWriteStream('output.txt'));

Clustering spawns multiple Node.js processes (workers) to utilize all CPU cores. Node.js is single-threaded, can't use multi-core by default. Cluster module: Master process forks workers, load balancer distributes requests. Example: const cluster = require('cluster'); const numCPUs = require('os').cpus().length; if(cluster.isMaster) { for(let i=0; i<numCPUs; i++) cluster.fork(); cluster.on('exit', (worker) => cluster.fork()); } else { app.listen(3000); } Benefits: Better CPU utilization, fault tolerance (if worker crashes, fork new one). Alternative: PM2 process manager handles clustering automatically. Use for: CPU-intensive tasks. Not needed for I/O-bound apps (single instance handles many connections efficiently).

Worker Threads enable true parallelism for CPU-intensive tasks (unlike Event Loop which is for I/O). Each worker has its own V8 instance, runs JavaScript in parallel. Use cases: Image processing, cryptography, data parsing. Example: const {Worker} = require('worker_threads'); const worker = new Worker('./worker.js', {workerData: {num: 5}}); worker.on('message', result => console.log(result)); worker.on('error', err => console.error(err)); // worker.js: const {parentPort, workerData} = require('worker_threads'); parentPort.postMessage(heavyComputation(workerData.num)); Difference from cluster: Workers share memory (via SharedArrayBuffer), clusters are separate processes. Use Workers for CPU-bound tasks within single app.

Middleware functions execute during request-response cycle, have access to req, res, next. Chain: Request → middleware1 → middleware2 → route handler → Response. Types: 1) Application-level: app.use(), 2) Router-level: router.use(), 3) Error-handling: 4 parameters (err, req, res, next), 4) Built-in: express.json(), express.static(), 5) Third-party: cors(), helmet(). Example: const logger = (req, res, next) => { console.log(`${req.method} ${req.url}`); next(); }; app.use(logger); Order matters: Auth middleware before routes. Call next() to continue chain. Middleware can: modify req/res, end cycle (res.send()), or pass error (next(err)).

Graceful shutdown: Clean up resources before process exits (close DB connections, finish pending requests). Implementation: process.on('SIGTERM', async () => { console.log('SIGTERM received, shutting down gracefully'); server.close(() => { console.log('HTTP server closed'); }); await db.close(); await redis.disconnect(); await jobQueue.close(); process.exit(0); }); setTimeout(() => { console.error('Forced shutdown'); process.exit(1); }, 30000); // Force exit after 30s. Also handle SIGINT (Ctrl+C). In Kubernetes: Terminate pod → SIGTERM → 30s grace period → SIGKILL. Benefits: Prevent data corruption, complete in-flight requests, clean disconnect from services. Test: docker stop (sends SIGTERM), kubectl delete pod.

1) Unit Testing: Test individual functions/methods in isolation. Mock dependencies. Fast, many tests. Example: Test single function that validates email. Tools: Jest, Mocha, Chai. 2) Integration Testing: Test multiple components together (API + DB). Example: Test API endpoint that creates user in database. Tools: Supertest, TestContainers. 3) End-to-End (E2E): Test entire flow from user perspective. Example: User signup → email sent → login. Slower, fewer tests. Tools: Cypress, Playwright. 4) Load/Performance Testing: Test under high load. Tools: k6, Apache JMeter. Pyramid: Many unit tests, fewer integration, fewest E2E.

Example with Jest: // userService.js function createUser(data) { if(!data.email) throw new Error('Email required'); return db.users.create(data); } // userService.test.js const userService = require('./userService'); const db = require('./db'); jest.mock('./db'); test('should create user with valid data', async () => { db.users.create.mockResolvedValue({id: 1, email: 'test@test.com'}); const user = await userService.createUser({email: 'test@test.com'}); expect(user).toHaveProperty('id'); expect(db.users.create).toHaveBeenCalledWith({email: 'test@test.com'}); }); test('should throw error without email', async () => { await expect(userService.createUser({})).rejects.toThrow('Email required'); }); Best practices: AAA pattern (Arrange, Act, Assert), test edge cases, mock external dependencies.

Use Supertest with Jest/Mocha: const request = require('supertest'); const app = require('./app'); describe('POST /api/users', () => { test('should create user', async () => { const res = await request(app).post('/api/users').send({email: 'test@test.com', password: '123456'}); expect(res.statusCode).toBe(201); expect(res.body).toHaveProperty('id'); expect(res.body.email).toBe('test@test.com'); }); test('should return 400 for invalid email', async () => { const res = await request(app).post('/api/users').send({email: 'invalid', password: '123456'}); expect(res.statusCode).toBe(400); }); }); Setup: Use test database, seed data before tests, cleanup after. Tools: TestContainers for Docker DB instances.

TDD: Write tests before implementation. Red-Green-Refactor cycle: 1) Red: Write failing test for desired functionality, 2) Green: Write minimal code to pass test, 3) Refactor: Improve code while keeping tests passing. Example: Test: expect(add(2, 3)).toBe(5); Implementation: function add(a, b) { return a + b; } Benefits: Better design (testable code), fewer bugs, documentation via tests, confidence in refactoring. Challenges: Slower initial development, requires discipline. Use for: Critical business logic, libraries. Not always practical for rapid prototyping or UI-heavy work.

Mocking isolates code being tested from external dependencies (DB, APIs, file system). Methods: 1) Jest mocks: jest.mock('./module'); module.function.mockReturnValue(value); 2) Sinon stubs: const stub = sinon.stub(object, 'method').returns(value); 3) Dependency injection: Pass dependencies as parameters. Example: // Instead of: function getUser(id) { return db.query('SELECT * FROM users WHERE id=?', id); } // Use: function getUser(id, database) { return database.query('SELECT * FROM users WHERE id=?', id); } // Test: const mockDb = { query: jest.fn().mockResolvedValue({id: 1, name: 'John'}) }; const user = await getUser(1, mockDb); Benefits: Fast tests (no actual DB calls), predictable results, test error scenarios.

1) Request metrics: Requests per second (RPS), response time (p50, p95, p99 percentiles), error rate (4xx, 5xx), 2) System metrics: CPU usage, memory usage, disk I/O, network I/O, 3) Application metrics: Active connections, queue size, cache hit ratio, 4) Database metrics: Query time, connection pool usage, slow queries, 5) Business metrics: Signups, orders, revenue. Tools: Prometheus (metrics collection), Grafana (visualization), Datadog, New Relic. Example alert: If p95 response time > 1s for 5 min, trigger alert. SLIs (Service Level Indicators): availability (99.9% uptime), latency (95% requests < 200ms).

Structured logging uses consistent, machine-readable format (JSON) instead of plain text. Traditional: 'User John logged in at 2025-01-31 10:30:00'. Structured: {timestamp: '2025-01-31T10:30:00Z', level: 'info', message: 'User logged in', userId: 123, username: 'John', ip: '192.168.1.1'}. Benefits: Easy to query (find all errors for user 123), aggregation (average response time), alerting. Tools: Winston, Pino (Node.js), ELK Stack (Elasticsearch, Logstash, Kibana). Example: logger.info({userId: user.id, action: 'login', duration: 45}); Query: SELECT AVG(duration) WHERE action='login'. Include: timestamp, level, correlation ID (trace requests across services), context (user, request ID).

Distributed tracing tracks requests as they flow through multiple microservices. Problem: In monolith, single log shows full request. In microservices, request spans 5+ services - hard to debug. Solution: Trace ID assigned to request, propagated through all services. Each service logs with trace ID. Tools: Jaeger, Zipkin, OpenTelemetry. Example: Request → API Gateway (trace: abc123, span: 1) → Auth Service (trace: abc123, span: 2) → User Service (trace: abc123, span: 3). View in UI: See full request timeline, identify bottlenecks (Auth Service took 2s). Benefits: Debug distributed systems, performance optimization, dependency mapping. Implementation: Add trace ID to headers, log with trace context.

APM monitors application performance in real-time, identifies bottlenecks. Features: 1) Transaction tracing: Track slow requests end-to-end, 2) Error tracking: Capture exceptions with stack traces, 3) Database monitoring: Slow queries, N+1 problems, 4) External service monitoring: API calls, response times, 5) Real user monitoring (RUM): Frontend performance. Tools: New Relic, Datadog APM, Dynatrace, AppDynamics. Example: APM shows /api/users endpoint averaging 800ms, drilling down reveals 750ms spent in DB query. Fix: Add index, reduce to 50ms. Benefits: Proactive issue detection, faster debugging, capacity planning. Costs money but worth it for production apps.

Health checks verify service is running and healthy. Types: 1) Liveness: Is process running? Example: GET /health returns 200 OK. 2) Readiness: Is service ready to serve traffic? Example: Check DB connection, cache availability. Implementation: app.get('/health', async (req, res) => { const checks = { db: await checkDB(), cache: await checkCache(), uptime: process.uptime() }; const healthy = checks.db && checks.cache; res.status(healthy ? 200 : 503).json(checks); }); async function checkDB() { try { await db.query('SELECT 1'); return true; } catch(e) { return false; } } Use: Kubernetes liveness/readiness probes, load balancer health checks. Include: timestamp, version, dependencies status.

Connection health monitoring detects issues before they impact users. Strategies: 1) Periodic ping: setInterval(async () => { try { await db.query('SELECT 1'); logger.debug('DB health check passed'); } catch(error) { logger.error('DB health check failed', error); // Alert, trigger reconnection } }, 30000); 2) Connection pool metrics: const metrics = { total: pool.totalCount, idle: pool.idleCount, waiting: pool.waitingCount }; if(metrics.waiting > 5) logger.warn('Pool exhaustion', metrics); 3) Query timeout monitoring: const timeout = setTimeout(() => logger.error('Query timeout'), 5000); await db.query('SELECT * FROM users'); clearTimeout(timeout); 4) Slow query logging: if(duration > 1000) logger.warn('Slow query', {query, duration}); 5) External monitoring: Prometheus metrics, health endpoints. Health endpoint: app.get('/health', async (req, res) => { try { await db.query('SELECT 1'); res.json({status: 'healthy', db: 'connected'}); } catch(error) { res.status(503).json({status: 'unhealthy', db: 'disconnected'}); } });

CI/CD automates software delivery. CI (Continuous Integration): Automatically build and test code when pushed to repo. Prevents broken code from merging. Example: Push to GitHub → GitHub Actions runs tests → If pass, merge to main. CD (Continuous Deployment): Automatically deploy code to production after passing tests. Or Continuous Delivery: Ready to deploy, requires manual approval. Pipeline: Code push → Build → Test → Deploy to staging → Deploy to production. Tools: GitHub Actions, GitLab CI, Jenkins, CircleCI. Example workflow: on: push, jobs: test (run npm test), deploy (deploy to AWS). Benefits: Faster releases, fewer bugs, reliable deployments.

Docker packages applications with dependencies into containers (lightweight, isolated, portable). Dockerfile defines image: FROM node:18, WORKDIR /app, COPY package.json ., RUN npm install, COPY . ., CMD ['node', 'server.js']. Build: docker build -t myapp .. Run: docker run -p 3000:3000 myapp. Benefits: 1) Consistency (works on dev, staging, prod), 2) Isolation (dependencies don't conflict), 3) Easy scaling (spin up multiple containers), 4) Faster deployment. docker-compose for multi-container apps: services: app: build: ., ports: 3000:3000, db: image: postgres, environment: POSTGRES_PASSWORD=secret. Use: Development, testing, production.

Virtual Machine: Includes full OS, runs on hypervisor (VMware, VirtualBox). Heavy (GBs), slow boot (minutes), strong isolation. Docker Container: Shares host OS kernel, includes only app and dependencies. Lightweight (MBs), fast boot (seconds), process-level isolation. Analogy: VM is entire house, container is apartment in building (shared infrastructure). Example: 10 VMs need 10 OS instances (10GB+ RAM). 10 containers share 1 OS (minimal overhead). Use VMs for: Different OS (Windows on Linux host), strong isolation. Use Containers for: Microservices, rapid scaling, resource efficiency. Can combine: Run Docker inside VMs for extra isolation.

Kubernetes (K8s) is container orchestration platform for managing containerized applications at scale. Features: 1) Auto-scaling: Add containers based on CPU/memory, 2) Load balancing: Distribute traffic, 3) Self-healing: Restart failed containers, 4) Rolling updates: Zero-downtime deployments, 5) Service discovery, 6) Secrets management. Components: Pods (group of containers), Deployments (desired state), Services (networking), Ingress (routing). When needed: Many microservices (10+), high availability requirements, dynamic scaling. Overkill for: Small apps, monoliths, startups. Alternatives: Docker Swarm (simpler), managed services (AWS ECS, Google Cloud Run). Learning curve steep but powerful for large-scale systems.

Environment variables store configuration outside code (DB credentials, API keys, feature flags). Example: process.env.DATABASE_URL, process.env.JWT_SECRET. Benefits: Different configs per environment (dev/staging/prod), security (no secrets in code), easy changes without redeployment. Management: 1) .env files (local): DB_HOST=localhost, DB_USER=admin. Load with dotenv: require('dotenv').config(). 2) CI/CD secrets: GitHub Secrets, GitLab Variables. 3) Cloud services: AWS Secrets Manager, Azure Key Vault. 4) Container orchestration: Kubernetes ConfigMaps/Secrets. Never commit .env to git! Use .gitignore. Validation: Ensure required vars exist at startup: if(!process.env.DATABASE_URL) throw new Error('DATABASE_URL required');

Blue-Green deployment strategy for zero-downtime releases. Two identical environments: Blue (current production), Green (new version). Process: 1) Deploy new version to Green, 2) Test Green thoroughly, 3) Switch router/load balancer to Green, 4) Blue becomes idle (backup), 5) If issues, switch back to Blue (rollback). Benefits: Zero downtime, instant rollback, test in production-like environment. Cons: Double resources (two environments), database migrations tricky (need backward compatibility). Example: AWS: Blue on EC2 instances 1-5, Green on 6-10. ELB switches from Blue to Green. Alternatives: Canary deployment (gradual rollout), rolling deployment (update instances one by one).

Event-Driven Architecture (EDA): Components communicate through events (state changes) rather than direct calls. Producer emits event, consumers react independently. Example: Order placed → Events: send_email, update_inventory, notify_shipping. Benefits: Loose coupling (services don't know about each other), scalability (add consumers without changing producer), asynchronous processing. Implementation: Event bus (Kafka, RabbitMQ, AWS EventBridge), event schema, event store. Patterns: 1) Event Notification: Lightweight, just signal something happened. 2) Event-Carried State Transfer: Include full data in event. 3) Event Sourcing: Store events as source of truth, rebuild state by replaying. Challenges: Eventual consistency, debugging (distributed traces), duplicate events (idempotency).

CQRS separates read (Query) and write (Command) operations into different models. Traditional: Single model for reads and writes. CQRS: Command model (optimized for writes), Query model (optimized for reads, often denormalized). Example: E-commerce: Command writes to normalized DB (orders, users, products). Query reads from denormalized view (order_details with all joined data). Benefits: Independent scaling (more read replicas), optimized queries, flexibility (different DBs for read/write). Implementation: Commands → Event Bus → Update Write DB → Publish Event → Update Read DB (eventual consistency). Use when: Complex domains, high read/write ratio difference. Overkill for simple CRUD apps.

WebSocket is full-duplex communication protocol over single TCP connection. Unlike HTTP (request-response), WebSocket allows bi-directional, real-time data flow. Handshake: HTTP upgrade request, then persistent connection. Use cases: 1) Chat applications (instant messaging), 2) Live notifications, 3) Real-time gaming, 4) Live dashboards (stock prices), 5) Collaborative editing. Example: const WebSocket = require('ws'); const wss = new WebSocket.Server({port: 8080}); wss.on('connection', ws => { ws.on('message', msg => wss.clients.forEach(client => client.send(msg))); }); HTTP vs WebSocket: Use HTTP for traditional request-response. Use WebSocket when server needs to push data to client frequently. Alternatives: Server-Sent Events (SSE) for one-way server→client.

Serverless: Run code without managing servers. Cloud provider handles infrastructure. Example: AWS Lambda (functions), Google Cloud Functions. Write function, upload, provider scales automatically. Characteristics: Event-driven, stateless, auto-scaling, pay-per-execution. Example: exports.handler = async (event) => { const user = await db.getUser(event.userId); return {statusCode: 200, body: JSON.stringify(user)}; }; Pros: No server management, auto-scaling, cost-effective (pay only when code runs), faster development. Cons: Cold starts (first request slow), vendor lock-in, limited execution time (15 min AWS Lambda), debugging harder, not suitable for long-running tasks. Use for: APIs, event processing, scheduled tasks, microservices.

Eventual Consistency: In distributed systems, all nodes will eventually have the same data, but not immediately. Contrast with strong consistency (immediate agreement). Example: Post on social media → Some users see it immediately, others see after few seconds. Acceptable when: Availability more important than immediate consistency (CAP theorem - choose AP over CP). Use cases: Social media feeds, product catalogs, analytics dashboards, DNS, CDN cache. Not acceptable: Financial transactions (need strong consistency), inventory management (can't oversell). Techniques: Conflict resolution (last-write-wins, vector clocks), read-repair, anti-entropy (background sync). Example: Cassandra (eventually consistent), DynamoDB (configurable).

Twelve-Factor App is methodology for building scalable, maintainable SaaS applications. Key factors: 1) Codebase: One codebase, many deploys, 2) Dependencies: Explicitly declare, 3) Config: Store in environment variables, 4) Backing services: Treat as attached resources, 5) Build, release, run: Strict separation, 6) Processes: Stateless, share-nothing, 7) Port binding: Self-contained, export via port, 8) Concurrency: Scale via process model, 9) Disposability: Fast startup/shutdown, 10) Dev/prod parity: Keep similar, 11) Logs: Treat as event streams, 12) Admin processes: Run as one-off processes. Benefits: Portability, cloud-native, scalability. Example: Instead of hardcoded DB URL, use process.env.DATABASE_URL. Widely adopted standard for modern apps.

Best practices: 1) Store in UTC: Always store timestamps in UTC in database. Example: created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP (UTC). 2) Convert on display: Convert to user's timezone in frontend/API response. 3) Use ISO 8601 format: 2025-01-31T10:30:00Z in APIs. 4) Libraries: moment-timezone, luxon, date-fns-tz. 5) Database: PostgreSQL TIMESTAMP WITH TIME ZONE vs TIMESTAMP (without). Use WITH TIME ZONE. 6) User timezone: Store user's timezone preference (America/New_York), not offset (offset changes with DST). Example: const utcDate = new Date(); const userDate = luxon.DateTime.fromJSDate(utcDate).setZone('America/New_York'); Common mistakes: Storing local time without timezone, using offsets instead of zones, not handling DST.

Feature flags (feature toggles): Enable/disable features without deploying code. Benefits: 1) Gradual rollout (10% users), 2) A/B testing, 3) Kill switch (disable broken feature), 4) Trunk-based development (merge incomplete code). Implementation: 1) Simple: Environment variable (FEATURE_NEW_CHECKOUT=true), check in code: if(process.env.FEATURE_NEW_CHECKOUT) { /* new code */ } 2) Database: Store flags in DB, cache in Redis. 3) External service: LaunchDarkly, Unleash, Firebase Remote Config. Example: const flags = await featureFlagService.getFlags(userId); if(flags.newCheckout) { /* new flow */ } else { /* old flow */ } Types: Release (temporary), Ops (circuit breaker), Permission (role-based), Experiment (A/B test). Best practice: Clean up old flags, don't overuse (code complexity).

Timeout: Prevent requests from hanging indefinitely. Example axios with timeout: axios.get('https://api.example.com/data', {timeout: 5000}); // 5 seconds. Handle timeout: catch (error) { if(error.code === 'ECONNABORTED') { /* handle timeout */ } }. Retry Logic: Automatically retry failed requests. Use exponential backoff to avoid overwhelming server. Example with axios-retry: axiosRetry(axios, { retries: 3, retryDelay: axiosRetry.exponentialDelay, retryCondition: (error) => axiosRetry.isNetworkOrIdempotentRequestError(error) || error.response.status === 503 }); Delay between retries: 1s, 2s, 4s (exponential). Add jitter (randomness) to prevent thundering herd. Only retry: Network errors, 5xx errors, 429. Don't retry: 4xx (client errors), non-idempotent operations. Circuit breaker pattern: Stop retrying if service consistently failing.

Logging best practices: 1) Middleware: app.use((req, res, next) => { const start = Date.now(); res.on('finish', () => { logger.info({ method: req.method, url: req.url, status: res.statusCode, duration: Date.now() - start, userId: req.user?.id, correlationId: req.correlationId }); }); next(); }); 2) Don't log: Sensitive data (passwords, tokens), full request bodies (PII, large payloads). 3) Sample large payloads: if(req.body) logger.debug({body: JSON.stringify(req.body).substring(0, 1000)}); 4) Structured logging (JSON): Easy to search, aggregate. 5) Log levels: ERROR (failures), WARN (deprecated endpoints), INFO (requests), DEBUG (detailed). 6) Include: Timestamp, correlation ID, user ID, IP, user-agent. 7) Centralized: Send to ELK, CloudWatch, Datadog. 8) Retention: Comply with regulations, rotate logs. Example: morgan with custom tokens.

DB connection can fail (network issues, DB restart, max connections). Retry logic ensures resilience. Implementation: async function connectWithRetry(maxRetries = 5, delay = 1000) { for(let i = 0; i < maxRetries; i++) { try { const connection = await db.connect(); logger.info('Connected to database'); return connection; } catch(error) { logger.warn(`DB connection attempt ${i + 1} failed: ${error.message}`); if(i === maxRetries - 1) throw error; // Exponential backoff with jitter await new Promise(resolve => setTimeout(resolve, delay * Math.pow(2, i) + Math.random() * 1000)); } } } Application startup: connectWithRetry().catch(err => { logger.error('Failed to connect to DB after retries', err); process.exit(1); }); Also handle connection loss during operation: pool.on('error', async (err) => { logger.error('Pool error', err); await connectWithRetry(); }); Best practices: Exponential backoff, jitter (prevent thundering herd), max retries, log attempts, fail fast if DB truly unreachable.

Both abstract data access, subtle differences. DAO (Data Access Object): Low-level, closely maps to database. One DAO per table. Methods mirror DB operations. Example: class UserDAO { findById(id) { return db.query('SELECT * FROM users WHERE id = ?', id); } insert(user) { return db.query('INSERT INTO users ...', user); } } Repository: Higher-level, domain-focused. One repository per aggregate root (DDD). Methods reflect business operations. Example: class UserRepository { findByEmail(email) { /* business logic, may query multiple tables */ } createUser(user) { /* validation, multi-table inserts */ } } DAO exposes: SQL concerns, low-level CRUD. Repository exposes: Domain language, business logic. In practice: Use DAO for simple CRUD, repository for complex domains with business rules. Modern frameworks (TypeORM, Sequelize): Repository pattern (findOne, save).