CI/CD Basics for Real Teams

1. Introduction

CI/CD is the practice of automatically building, testing, and deploying software every time code changes. It removes the manual steps between a developer committing code and that code running in production — reducing the risk of each change and making it possible to ship multiple times a day safely.

This guide explains how real teams structure their pipelines, what each stage does, where things typically go wrong, and how to think about CI/CD as an operational system that needs to be maintained, not just set up once and forgotten.

2. What CI/CD Means

Continuous Integration (CI) means that every code change is automatically built and tested as soon as it's pushed. The goal is to catch integration problems — two developers' changes breaking each other — quickly, while the code is fresh. Without CI, integration is a manual, slow, painful event that happens before releases.

Continuous Delivery (CD) means that every change that passes CI is automatically packaged and made deployable. With Continuous Deployment, passing changes are deployed to production automatically. Many teams do continuous delivery (always deployable) but trigger production deployments manually via approval gates.

3. Typical Pipeline Stages

Most CI/CD pipelines follow the same broad structure, even though the specific tools and configuration vary. Understanding the purpose of each stage helps you understand why failures in each stage matter:

• Trigger — a Git push, a merge request, or a scheduled run starts the pipeline
• Build — compile code, build a container image, or produce an artifact
• Test — run unit tests, integration tests, linting, and security scanning
• Package — create the final deliverable (Docker image, JAR, ZIP) and push it to a registry or artifact store
• Deploy — apply the change to an environment (dev, staging, production)
• Verify — run smoke tests or health checks after deployment to confirm the deployment succeeded

4. Build, Test, Package, Deploy in Practice

In a container-based workflow, the pipeline typically looks like this: code is pushed to Git, the CI system checks out the code, runs tests, builds a Docker image, pushes the image to a registry (ECR, Docker Hub, GitLab Container Registry), and then updates a Kubernetes deployment to use the new image tag.

# Typical GitLab CI pipeline structure (.gitlab-ci.yml):

stages:
  - test
  - build
  - deploy

test:
  stage: test
  script:
    - npm ci
    - npm test

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

deploy-staging:
  stage: deploy
  script:
    - kubectl set image deployment/my-app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - kubectl rollout status deployment/my-app --timeout=120s

The critical detail: always tag images with the Git commit SHA, not :latest. Immutable tags make rollbacks possible and make debugging much easier — you can always trace which version is running.

5. Runners and Agents

A runner (GitLab) or agent (Jenkins) is the machine that actually executes your pipeline jobs. In GitLab CI, jobs run inside Docker containers on runners. In Jenkins, agents can be Docker containers, VMs, or bare metal nodes. The runner's environment — its OS, installed tools, network access, resource limits, and Docker socket access — directly affects what your pipeline can do.

Common runner issues: the runner runs out of disk space (Docker image layers accumulate), the runner can't reach a private registry, the runner's Docker socket isn't accessible for Docker-in-Docker builds, or the runner hits a timeout. The Fix GitLab Runner Timeout guide covers the most common of these.

6. Environment Variables and Secrets

Pipeline steps need credentials: Docker registry passwords, Kubernetes cluster tokens, AWS access keys, API keys. These should never be hardcoded in your pipeline configuration. Every CI/CD platform has a mechanism for injecting secrets as environment variables at runtime — use it.

The most common mistake is storing secrets in the repository itself (even in a branch nobody looks at). The second most common mistake is scoping secrets incorrectly — a variable defined at group level in GitLab won't be available in a project unless it's configured correctly. The Fix Environment Variables Not Working in CI/CD guide covers the full diagnostic path.

# Safe pattern: reference secrets from CI variables, never hardcode
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

# Unsafe: never do this
- docker login -u myuser -p actualpassword registry.example.com

7. Common CI/CD Failure Patterns

Most pipeline failures fall into a small number of categories. Knowing how to recognise them saves significant time:

• Tests fail — read the test output, not just the pipeline summary. The actual assertion failure is buried in the log.
• Docker build fails — authentication, wrong base image tag, COPY path issues. See Fix Docker Build Failed in CI/CD.
• Pipeline hangs — usually a command waiting for input that will never come. Add -y or --non-interactive flags to all interactive commands.
• Missing environment variable — the command runs but uses an empty string instead of failing loudly. Always validate required variables at the start of the job.
• Deployment succeeds but the app is broken — the pipeline didn't verify the deployment. Always run a post-deploy health check.

For a systematic debugging approach, see Fix CI/CD Pipeline Failed.

8. Practical Team Workflow

A pipeline is only useful if the team actually uses it. The practices that make CI/CD work well in practice:

• Keep the main branch always deployable. If a failing test blocks deployment, fix it immediately — don't let it sit.
• Keep the pipeline fast. Cache dependencies. Parallelise tests. A slow pipeline that nobody waits for is useless.
• Make failures loud and specific. The error message should tell you exactly what failed and where to look. If it doesn't, add better logging.
• Test environments should match production. If staging uses a different database version or different secrets structure, test failures in staging won't reflect production behaviour.
• Deploy to staging automatically, production manually. Most teams benefit from automated staging deployments but human-approved production deployments. Adjust based on your risk tolerance.

9. Where to Go Next

The CI/CD category covers the operational side of pipelines in depth — the specific failures you'll encounter and how to fix them. Once you have a working pipeline, the next level is understanding deployment strategies (blue-green, canary, rolling updates) and integrating automated security scanning and infrastructure-as-code validation into the pipeline.