Shipping Fintech on ECS Without a Dedicated Platform Team

Building a release system around evidence, constrained credentials, and recoverable change

Abstract

A small team does not need a large internal platform to ship safely. It does need a release path that makes the risky decisions explicit.

This paper describes the deployment system I built around a React and Express fintech application on Amazon ECS Fargate. The application runs as one deployable container with Postgres, an Application Load Balancer, provider integrations, and a small companion rule-engine service. GitHub Actions owns tagged releases; Terraform owns the infrastructure shape.

The useful lesson is not “use ECS.” It is how the release path was organized: environment-matching release identities, preflight checks, short-lived cloud credentials, validation before cutover, migrations inside the runtime network, health-aware rolling replacement, connection draining, post-deploy smoke tests, and an explicit last-known-good rollback target.

The platform is the sequence of evidence and constraints, not the number of tools.

1. Start with the operating constraint

The application did not have a dedicated platform team. It also did not need a Kubernetes adoption project.

Its production shape was modest:

  • one React single-page application;
  • one Express API;
  • one Postgres database;
  • one application container;
  • one small Python rule-engine service;
  • Stripe, Quiltt, and SES integrations;
  • staging and production environments;
  • a lean delivery team.

That simplicity did not make deployment trivial. Billing state, account data, sessions, schema migrations, webhook processing, and planning workflows made a bad release consequential.

The central platform question was therefore:

What is the smallest release system that can prove the artifact, protect the database, preserve service availability, and leave a usable recovery path?

The answer was a staged deployment contract rather than a larger orchestration stack.

2. Give every release an immutable identity

The deployment workflow accepts an explicit environment and release tag:

  • stg-v* for staging;
  • prd-v* for production.

A manual deployment is rejected unless the tag prefix matches the target environment. The workflow resolves the tag to a commit and records that commit in the run summary.

This removes a surprisingly common ambiguity:

What code are we actually deploying?

Branches move. A manual workflow that accepts an arbitrary branch or unvalidated string can deploy a different commit than the operator intended. A release tag is not perfect change control, but it provides a stable name, an inspectable commit, and a straightforward rollback target.

The container image uses the release tag, and an existing image for that tag is reused rather than silently overwritten. That makes “redeploy the release” mean the same artifact, not a new build under an old name.

3. Fail before acquiring production power

The first workflow job is a configuration preflight.

It checks:

  • target environment;
  • matching release tag;
  • AWS region and role;
  • ECR repository;
  • ECS cluster, service, and container names;
  • deployment base URL;
  • task-definition path;
  • task subnets and security group;
  • required public build configuration;
  • database-secret reference.

If the contract is incomplete, the workflow stops before building or modifying cloud resources.

This distinction matters. A deployment job should not discover halfway through that it lacks the database secret, task subnet, or service name. By that point it may already have published an image, registered a partial task definition, or changed infrastructure state.

Preflight checks are cheap and deterministic. Put them before privileged work.

4. Use short-lived deployment credentials

The workflow uses GitHub Actions OpenID Connect to assume an AWS role. It does not store a long-lived AWS access key in the repository.

GitHub describes this model as exchanging a workflow identity token for a short-lived cloud credential scoped by the provider’s trust policy.1 GitHub also documents how that workflow identity is configured for AWS trust.2

The workflow must request id-token: write, but that permission only allows it to request the identity token; AWS still decides what the assumed role may do.

The operational benefits are concrete:

  • no static AWS credential to rotate in GitHub;
  • the token expires with the job;
  • trust can be restricted by repository, environment, branch, or other claims;
  • deployment permissions stay in IAM;
  • staging and production can use different roles and approval rules.

OIDC is not automatically least privilege. A broad AWS role remains broad. But it removes one class of durable secret and makes the trust boundary inspectable.

5. Validate the release artifact before cutover

The release workflow runs application validation before the deployment job can start.

The gate includes:

  • TypeScript checking;
  • application route-map drift detection;
  • focused unit tests for the cash-flow foundation;
  • a clean database schema bootstrap;
  • application startup and health wait;
  • Node end-to-end suites;
  • browser-based Playwright flows;
  • a narrower production release smoke.

Staging runs the broader onboarding and cash-flow suites. Production runs a focused path intended to prove the release artifact without depending on live provider mutations.

This is a deliberate asymmetry. Staging should absorb broad behavior validation. Production pre-cutover validation should be fast enough to run every time and specific enough to catch release-critical regressions.

The gate also uploads logs and browser artifacts even when it fails. A red job without evidence is only a delay. A red job with server logs, screenshots, and traces is a diagnostic tool.

6. Run migrations where the application runs

The production database is not reachable directly from the public GitHub runner. That is a feature.

Schema migrations run as a one-off ECS task:

  1. render the new task definition;
  2. register it with the release image and runtime configuration;
  3. start a standalone task with the migration command;
  4. run it in the application network;
  5. wait for the task to stop;
  6. inspect the container exit code;
  7. update the service only if the migration succeeded.

AWS supports standalone ECS tasks for one-time work such as batch processes.3 Using the release image for migrations gives the task the same code, network reachability, and secret references as the application revision.

More importantly, a migration failure happens before traffic moves. The old service continues serving on its existing task definition.

This does not make every migration reversible. A destructive schema change can still make application rollback unsafe. The release discipline is therefore:

  • prefer additive, backward-compatible migrations;
  • stop before cutover on migration failure;
  • treat data repair as explicit work;
  • do not pretend that redeploying an old image reverses an irreversible schema change.

7. Keep infrastructure ownership and release ownership separate

Terraform owns:

  • VPC and subnet shape;
  • security groups;
  • RDS;
  • ALB and target groups;
  • ECS clusters and service scaffolds;
  • ECR;
  • logs;
  • runtime parameter and secret references;
  • DNS and certificate resources;
  • IAM roles.

The tagged GitHub workflow owns:

  • application image build;
  • image publication;
  • task-definition revision;
  • migration execution;
  • ECS service update;
  • release verification.

The ECS service resource ignores task-definition drift after Terraform creates the scaffold. Without that boundary, a routine Terraform apply could roll the service back to the bootstrap image.

This is an example of two tools cooperating through an explicit ownership rule. The alternative is not “one tool is cleaner.” The alternative is two tools both believing they own the current release.

8. Make rolling replacement match the replica count

The service initially runs one desired application task. That makes the ECS deployment percentages easy to underestimate.

With:

desired count = 1

minimum healthy percent = 100

maximum percent = 200

ECS can start one replacement task while keeping the current task healthy. It cannot intentionally stop the only healthy task merely to make room for the new revision.

AWS documents minimumHealthyPercent as the lower bound on healthy running tasks during a rolling deployment and maximumPercent as the upper bound on total running tasks.4

The numbers are not an availability guarantee. The cluster still needs capacity, the new task must become healthy, and a single steady-state replica still leaves other failure modes. But changing the minimum from 50 to 100 closed a concrete rollout gap for a one-replica service.

Configuration should be evaluated against actual counts, not only percentages.

9. Health is necessary; draining completes the handoff

The container, ALB target group, and workflow each use health checks.

That proves the replacement can answer requests. It does not prove the old task can stop without interrupting work already in flight.

The rollout currently has two of the three behaviors needed for a complete handoff:

  1. ECS waits for the replacement task to become healthy.
  2. The ALB stops sending new traffic to the old target and allows connection draining.

AWS notes that deregistering targets enters a draining state so in-flight requests can complete; terminating a target before draining finishes can produce 500-level errors.5

The missing third behavior is application cooperation. In the implementation snapshot used for this paper, the Node process does not install an explicit SIGTERM handler to stop accepting new work and close HTTP and Socket.IO resources within a bounded window.

ALB draining reduces risk, but it does not prove graceful application shutdown. That gap is a useful reminder: graceful deployment is not an infrastructure setting. The load balancer, scheduler, and process must agree on the shutdown protocol. Adding and testing that process-level behavior remains a release hardening item.

10. Treat secret drift as a release failure

Managed database credentials introduce a subtle operational failure. RDS can rotate or replace a credential while the application’s assembled DATABASE_URL secret still points at the old value.

The deploy path refreshes the application secret from the live RDS master secret, then runs an explicit drift check before the new task definition is promoted.

That turns credential synchronization from tribal knowledge into release logic.

The broader pattern applies to any derived secret:

  • identify the authoritative source;
  • materialize the application-facing representation;
  • compare before cutover;
  • fail with a bounded diagnostic when they diverge;
  • do not print the secret while proving equality.

11. Verify production after deployment

A successful ECS update is not proof that the product works.

After the service reaches stability, the release path verifies:

  • public and API health endpoints;
  • canonical domain behavior;
  • landing and authentication entry;
  • authenticated session behavior;
  • protected application access;
  • provider configuration health;
  • rule-engine reachability;
  • relevant support and audit visibility.

CloudWatch logs are checked for application, migration, rule-engine, and webhook failures before sign-off.

This distinction changed how I think about deployment status:

image pushed != task registered

task registered != service stable

service stable != application healthy

application healthy != critical workflow verified

workflow verified != release understood

Every equality sign that does not hold needs its own proof.

12. Keep rollback boring

The rollback target is the previous known-good release tag and task-definition revision.

The procedure is intentionally uncreative:

  • stop promotion when a pre-cutover gate fails;
  • leave the previous service revision serving;
  • for post-cutover failure, redeploy the last known-good image or task definition;
  • inspect schema compatibility before rollback;
  • revert workflow or Terraform wiring separately from application behavior;
  • verify the recovered environment with the same smoke path.

Rollback documentation should name both the artifact and the evidence that makes it “known good.” Otherwise the team is choosing an older unknown under pressure.

13. What I would add as the system grows

The current path is appropriate to the current scale, not a final platform.

The next investments would be triggered by observed risk:

  • multiple steady-state replicas when availability requirements justify the cost;
  • ECS deployment circuit breaker and alarm-based rollback;
  • stronger release provenance and artifact signing;
  • automated migration compatibility checks;
  • blue/green traffic shifting if version isolation becomes more valuable than rolling simplicity;
  • synthetic production monitoring beyond deployment time;
  • formal recovery-time and recovery-point tests.

The rule is to add a control in response to a named failure mode. Platform complexity without a threat or reliability model is just another production system to operate.

14. Practical design rules

  1. Give each deployment an immutable release identity.
  2. Validate environment and configuration before acquiring cloud permissions.
  3. Prefer short-lived federated credentials over stored deployment keys.
  4. Run migrations from the release artifact inside the runtime network.
  5. Stop before service cutover when migration or validation fails.
  6. Define whether Terraform or the release workflow owns each mutable resource.
  7. Evaluate ECS deployment percentages against the real replica count.
  8. Combine scheduler health, load-balancer draining, and application shutdown.
  9. Treat derived-secret drift as a release concern.
  10. Verify the live product separately from the deployment mechanism.
  11. Keep the last-known-good artifact explicit and test the rollback path.
  12. Add platform machinery only when it answers a concrete operating risk.

Conclusion

A lean team can operate a consequential application without first becoming a platform organization.

The requirement is not a particular orchestrator. It is a release system that narrows ambiguity: which artifact, which environment, which credentials, which schema, which health signal, which smoke test, and which recovery target.

ECS, Terraform, GitHub Actions, and Postgres supplied the mechanisms in this case. The engineering work was connecting them into a sequence where unsafe change stops early and successful change leaves evidence.

That sequence is the platform.

References

Footnotes

  1. GitHub Docs, “OpenID Connect”. ↩

  2. GitHub Docs, “Configuring OpenID Connect in Amazon Web Services”. ↩

  3. Amazon Web Services, “Amazon ECS standalone tasks”. ↩

  4. Amazon Web Services, “Deploy Amazon ECS services by replacing tasks”. ↩

  5. Amazon Web Services, “Edit target group attributes for your Application Load Balancer”. ↩

Citation sources

  1. GitHub Actions OpenID Connect documentation · Return to citation
  2. GitHub OIDC configuration for AWS · Return to citation
  3. Amazon ECS standalone-task documentation · Return to citation
  4. Amazon ECS rolling-deployment documentation · Return to citation
  5. AWS Application Load Balancer draining documentation · Return to citation

Related project