AWS (ECS Fargate)

One CloudFormation stack deploys DataFactory into an existing VPC in your AWS account. The VPC can be fully private, with no internet gateway and no NAT. The stack creates:

• ECS Fargate services: web, API (×2), Prune worker and Bloom worker;
• an internal Application Load Balancer, HTTPS only;
• RDS PostgreSQL (encrypted, 14-day backups, deletion protection);
• ElastiCache Redis (TLS, AUTH token, encrypted at rest);
• EFS for generated files (encrypted, AWS Backup on);
• Secrets Manager secrets, CloudWatch Logs and least-privilege IAM roles;
• optionally, the VPC endpoints the services need.

What you need

• An AWS account and region, and a role that can create the resources above.
• A VPC with two private subnets in different Availability Zones (and their route tables, if the stack creates the S3 endpoint).
• An ACM certificate for the host name users will browse to. ACM Private CA works.
• A way to point that host name at the load balancer: a CNAME or a Route 53 alias.
• A workstation with the AWS CLI v2, openssl, and Docker or skopeo, that can reach ECR. A bastion inside the VPC is fine.
• The DataFactory bundle, extracted, and your licence file.

VPC endpoints. Without internet access, the tasks need ecr.api, ecr.dkr, logs and secretsmanager (interface endpoints) and s3 (gateway endpoint). Set CreateVpcEndpoints=true to have the stack create them, or false if your VPC already has them.

1. Copy the images into ECR

From the extracted bundle:

./push-images.sh --ecr 123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/datafactory

It checks the bundle, creates the repositories (immutable tags, scan on push) and copies the images. ECS needs only datashrink-be and datashrink-fe.

2. Create the application secret

./aws/create-secrets.sh -r ap-southeast-2 -l /path/acme.licence

This creates the secret datafactory/app with the session signing key, the credential encryption key, the first admin sign-in and the licence. It prints the secret's ARN and the admin password.

Never rotate datafactory/app

Keep it as long as the data exists. Saved database passwords can't be read without its CREDENTIALS_ENCRYPTION_KEY, and RDS snapshots are useless without it.

3. Create the stack

aws cloudformation deploy \
  --region ap-southeast-2 \
  --stack-name datafactory \
  --template-file aws/datafactory.yaml \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides \
      VpcId=vpc-0abc \
      PrivateSubnet1=subnet-0aaa PrivateSubnet2=subnet-0bbb \
      PrivateRouteTableIds=rtb-0ccc,rtb-0ddd \
      AllowedCidr=10.0.0.0/8 \
      ImageRepositoryPrefix=123456789012.dkr.ecr.ap-southeast-2.amazonaws.com/datafactory \
      ImageTag=1.3.1 \
      AppSecretArn=arn:aws:secretsmanager:ap-southeast-2:123456789012:secret:datafactory/app-AbCdEf \
      PublicHostName=datafactory.example.com \
      CertificateArn=arn:aws:acm:ap-southeast-2:123456789012:certificate/0000-...

The first create takes about 20 minutes, mostly RDS and ElastiCache. Then:

1. Point PublicHostName at the stack's LoadBalancerDnsName output.
2. Browse to https://<PublicHostName> and sign in as admin with the password from step 2. Change it.
3. On your databases, allow inbound access from the stack's AppSecurityGroup output.

Parameter	Default	Notes
LoadBalancerScheme	internal	internet-facing needs public subnets; not recommended
AllowedCidr	10.0.0.0/8	Who can reach the load balancer
CreateVpcEndpoints	true
ApiCount, WebCount, BloomWorkerCount	2, 2, 1	The Prune worker always runs exactly 1 task
DbInstanceClass, DbAllocatedStorage, DbMultiAz	db.t4g.medium, 50 GB, false	Use Multi-AZ for production
CacheNodeType	cache.t4g.small
AiProvider, AiBaseUrl, AiModel	none	Internal model endpoints only on an isolated network
ConnectionAllowedHosts	(any)	Restrict which database hosts users can connect to