Manual Setup

Manual deployment + advanced configuration

Advanced deployment guide for users who need direct Terraform control, CI/CD integration, or custom infrastructure configurations.

Prerequisites

Required Software

Terraform (>= 1.0) - Installation Guide
AWS CLI (>= 2.15.0) - Installation Guide
Python (>= 3.13) - Installation Guide
uv - Installation Guide
Node.js (>= 24.2.0) & npm (>= 11.4.2) - Installation Guide

AWS Account Requirements

AWS account with appropriate permissions
AWS CLI configured: aws configure
TwelveLabs API Key - Get your API key

Manual Deployment Process

Clone and Prepare Repository

git clone https://github.com/kubrick-ai/kubrick.git
cd kubrick

Build Lambda Packages

Build the serverless functions and layers manually with
Terminal window
```
./lambda/build-package.sh
```
or use your own CI/CD solution.
Initialize Terraform

Navigate to the Terraform directory and initialize:
Terminal window
```
cd terraform
terraform init
```
Configure Variables

Create your terraform.tfvars file based on the example:
Terminal window
```
cp terraform.tfvars.example terraform.tfvars
```
Edit terraform.tfvars with your configuration:
```
# Required variables
aws_region = "us-east-1"  # Your AWS region

# Database credentials
db_username = "postgres"  # Your PostgreSQL username
db_password = "your-secure-password"  # Your PostgreSQL password

# API keys
twelvelabs_api_key = "your-twelvelabs-api-key"  # Your TwelveLabs API key

# Optional: Override defaults
aws_profile = "default"
secret_name = "kubrick_secret"
stage_name = "v1_0"
```
Variable Reference
All available variables are defined in variables.tf:
- aws_region - AWS region for deployment (default: us-east-2)
- secret_name - AWS Secrets Manager secret name (default: kubrick_secret)
- db_username - PostgreSQL username (required)
- db_password - PostgreSQL password (required, sensitive)
- twelvelabs_api_key - TwelveLabs API key (required, sensitive)
- aws_profile - AWS CLI profile (default: default)
- stage_name - API Gateway stage (default: v1_0)
Review Deployment Plan

Generate and review the execution plan:
Terminal window
```
terraform plan
```
This shows all resources that will be created, modified, or destroyed.
Deploy Infrastructure

Apply the Terraform configuration:
Terminal window
```
terraform apply
```
Type yes when prompted. Deployment takes 10-15 minutes.

Working with Existing AWS Resources

Importing Existing Secrets

If you already have a secret named kubrick_secret in AWS Secrets Manager:

Verify Secret Contents

Check your existing secret contains required keys:
Terminal window
```
aws secretsmanager get-secret-value \
  --secret-id kubrick_secret \
  --query SecretString --output text
```
Required keys:
- DB_USERNAME
- DB_PASSWORD
- TWELVELABS_API_KEY

Update Secret if Needed

If keys are missing or have different names:

aws secretsmanager update-secret \
  --secret-id kubrick_secret \
  --secret-string '{
    "DB_USERNAME": "postgres",
    "DB_PASSWORD": "your-password",
    "TWELVELABS_API_KEY": "your-api-key"
  }'

Import Existing Secret

Import the secret into Terraform state:

terraform import module.secrets_manager.aws_secretsmanager_secret.kubrick_secret kubrick_secret

Verify Import
Terminal window
```
terraform plan
```

Handling Resource Conflicts

Common Import Issues

ResourceExistsException: Follow the secret import steps above
VPC/Subnet conflicts: Ensure your AWS account doesn’t have conflicting default VPC settings
IAM role conflicts: Check for existing roles with similar names

Terraform Architecture Overview

Core Modules

api_gateway - REST API endpoints for video operations
cloudfront - CDN for global content delivery
dynamodb - Embedding cache for performance optimization
iam - Roles and policies for service permissions
lambda - Serverless functions and layers
rds - PostgreSQL database for metadata
s3 - Storage buckets for videos and static assets
s3_notifications - Event triggers for video processing
secrets_manager - Secure credential storage
sqs - Message queues for async processing
vpc_network - Network infrastructure and security

Lambda Functions Deployed

API Handlers:
- api_fetch_tasks_handler - Task status and management
- api_fetch_videos_handler - Video listing and metadata
- api_search_handler - Semantic search with embeddings
- api_video_upload_link_handler - Presigned upload URLs
Processing Functions:
- db_bootstrap - Database initialization
- s3_delete_handler - Cleanup on video deletion
- sqs_embedding_task_consumer - Process embedding jobs
- sqs_embedding_task_producer - Create embedding jobs
Shared Layers:
- config_layer - Common configuration utilities
- embed_service_layer - TwelveLabs API integration
- response_utils_layer - HTTP response formatting
- s3_utils_layer - S3 operation utilities
- vector_database_layer - Vector similarity operations

Verification and Testing

After deployment completes:

Check Terraform Output

Review important outputs:
Terminal window
```
terraform output
```
Key outputs include:
- CloudFront distribution URL
- API Gateway endpoint
- S3 bucket names
- RDS endpoint

Verify AWS Resources

Check resource creation:

# Lambda functions
aws lambda list-functions --query 'Functions[?contains(FunctionName, `kubrick`)]'

# API Gateway
aws apigateway get-rest-apis --query 'items[?contains(name, `kubrick`)]'

# S3 buckets
aws s3 ls | grep kubrick

Test the Playground

Access the CloudFront URL from the Terraform output to test the web interface.
API Health Check

Test API endpoints:
Terminal window
```
curl https://your-api-gateway-url/v1_0/videos
```

Customization Options

Custom AWS Region

Update aws_region in your terraform.tfvars:

aws_region = "eu-west-1"  # Europe (Ireland)

Custom Database Configuration

Modify database settings in terraform.tfvars:

db_username = "kubrick_admin"
db_password = "your-complex-password-here"

API Gateway Stage Names

Change the API version/stage:

stage_name = "production"  # Creates /production/ endpoints

Troubleshooting

Common Issues

State Management

# Backup state
terraform state pull > kubrick.tfstate.backup

# List resources in state
terraform state list

# Remove problematic resource
terraform state rm aws_s3_bucket.example

Cleanup and Management

Destroying Infrastructure

terraform destroy

Selective Resource Management

Remove specific resources:

# Remove CloudFront
terraform destroy -target=module.cloudfront

# Remove RDS
terraform destroy -target=module.rds

Advanced Configuration

CI/CD Integration

Example GitHub Actions workflow:

name: Deploy Kubrick
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
      - name: Terraform Deploy
        run: |
          cd terraform
          terraform init
          terraform apply -auto-approve
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          TF_VAR_twelvelabs_api_key: ${{ secrets.TWELVELABS_API_KEY }}

Multi-Environment Setup

Create separate .tfvars files for each environment:

# Development
terraform apply -var-file="dev.tfvars"

# Production
terraform apply -var-file="prod.tfvars"

IaC File Structure

Directorykubrick/
- …
- Directoryterraform/
  - Directorymodules/
    Directoryapi_gateway/
    …
    Directorycloudfront/
    …
    Directorydynamodb/
    …
    Directoryiam/
    …
    Directorylambda/
    …
    Directoryrds/
    …
    Directorys3/
    …
    Directorys3_notifications/
    …
    Directorysecrets_manager/
    …
    Directorysqs/
    …
    Directoryvpc_network/
    …
  - main.tf
  - variables.tf
  - terraform.tfvars.example