NucleusCloud
Cloud-native bioinformatics VCF variant processing pipeline on AWS. Accepts genomic VCF files via presigned S3 upload, processes them asynchronously through SQS/SNS-driven microservices, tiers cold storage to Glacier based on subscription level, and emails results via SES - all behind Globus Auth and Stripe billing. Entire stack deployed from a single CloudFormation template.
Architecture
How it works
The Flask app on EC2 Web authenticates researchers through Globus Auth - a federated identity system trusted by institutions like NIH and XSEDE. On login, Stripe resolves the user's subscription tier: Basic (150KB VCF upload limit, aggressive 5-minute Glacier archival) or Premium (unlimited file sizes, indefinite hot S3 preservation). Job state is tracked in DynamoDB throughout.
After enforcing the per-tier size limit, Flask generates a presigned S3 URL server-side. The VCF file bytes travel directly from the browser to S3 - the EC2 Web instance never handles file data. This keeps the web server free for API requests regardless of file size, and S3 handles durability and replication automatically.
The S3 upload event enqueues a message in SQS. The VCF Analyzer worker on EC2 Ann polls this queue, picks up the job, and processes the genomic variant file - running analysis and annotation logic from pipeline/. Processed results are written back to S3, and job status is updated in DynamoDB. EC2 Ann scales independently of the web tier.
Completion triggers an SNS message that fans out to two microservices on EC2 Util. The Archiver receives the SNS event and migrates Basic-tier results to S3 Glacier within 5 minutes; Premium results stay in hot S3. The Notifier polls a separate SQS queue and emails the researcher via SES on success or failure - completely decoupled from the web and analysis tiers.
In parallel, SNS triggers a Step Functions state machine for structured archival - but only for objects that meet the Basic-tier lifecycle policy. Premium-tier objects bypass this flow entirely and remain in hot S3 indefinitely. For Basic-tier objects, the three-stage machine runs: (1) Validate - checksum verification and format validation, (2) Process - indexing and metadata extraction, (3) Archive - cold storage migration to S3 Glacier within 5 minutes. Each stage is independently retryable with exponential backoff.
When a researcher requests a previously archived file, the Retriever on EC2 Util polls SQS for retrieval jobs and initiates a Glacier batch retrieval. Once Glacier thaws the data, the Lambda Restorer function fires - it copies the restored bytes back to hot S3 and returns a presigned download URL. The two-service split keeps long-running Glacier polling off Lambda's execution limits.
All three EC2 instances, IAM roles, S3 buckets, SQS queues, SNS topics, Lambda, Step Functions, DynamoDB tables, and VPC networking are defined in a single CloudFormation template.yaml. LocalStack mirrors the full AWS environment locally via start_all_local.sh. Moto mocks every AWS service call in unit tests, giving full coverage without cloud spend.