Monday, 23 March 2026

Apache Airflow: Force Start / Trigger Job (DAG)

 

Apache Airflow: Force Start / Trigger Job (DAG)

Introduction

Apache Airflow is an open-source workflow orchestration tool used to programmatically author, schedule, and monitor workflows using Directed Acyclic Graphs (DAGs).

Unlike traditional schedulers (Autosys, Control-M), Airflow doesn’t use the exact term “force start”, but it provides similar functionality through manual triggering of DAGs.


What is Force Start in Airflow?

In Airflow, Force Start = Manually Triggering a DAG run, ignoring:

  • The defined schedule (schedule_interval)
  • Previous run dependencies (if configured)
  • Timing constraints

It allows you to execute a workflow immediately.


Methods to Force Start a DAG


1. Using Airflow UI

Steps:

  1. Open Airflow Web UI
  2. Go to DAGs
  3. Find your DAG
  4. Click the ▶ Trigger DAG button

This creates a new DAG run instantly.


2. Using CLI (Command Line)

airflow dags trigger <dag_id>

Example:

airflow dags trigger data_pipeline_dag

This will:

  • Create a new DAG run
  • Start execution immediately

3. Trigger with Custom Configuration

You can pass parameters during trigger:

airflow dags trigger <dag_id> --conf '{"key":"value"}'

Example:

airflow dags trigger etl_dag --conf '{"run_type":"manual"}'

4. Trigger Specific Execution Date

airflow dags trigger <dag_id> --exec-date 2026-03-23T10:00:00

Useful for:

  • Backfilling
  • Re-running specific time windows

Force Running Individual Tasks

Sometimes you don’t want to run the full DAG, only specific tasks.

Mark Task as Success (Skip Dependencies)

In UI:

  • Select task → Mark Success

Clear Task (Re-run Task)

airflow tasks clear <dag_id> -t <task_id>

This:

  • Resets task state
  • Triggers re-execution

Ignoring Dependencies

Airflow provides options to bypass dependencies:

airflow tasks run <dag_id> <task_id> --ignore-dependencies

Other useful flags:

  • --ignore-all-dependencies
  • --ignore-depends-on-past

When to Use Force Start

1. Testing

  • Validate DAG logic
  • Debug task failures

2. Recovery

  • Re-run failed pipelines
  • Skip broken upstream tasks

3. Ad-hoc Runs

  • Run pipelines outside schedule

Important Considerations

⚠️ Dependency Handling

  • Forcing tasks may break DAG logic

⚠️ Data Consistency

  • Running out of order can lead to incorrect results

⚠️ Duplicate Runs

  • Manual triggers can create overlapping executions

Monitoring DAG Runs

UI:

  • Graph View
  • Tree View
  • Gantt View

CLI:

airflow dags list-runs -d <dag_id>

Comparison with Autosys & Control-M

FeatureAutosysControl-MAirflow
Force Start Commandsendeventctmorderairflow dags trigger
UnitJobJobDAG
Dependency ControlIgnoredIgnoredConfigurable
UI TriggerLimitedYesYes

Best Practices

  • Avoid frequent manual triggers in production
  • Use parameters (--conf) for controlled runs
  • Monitor logs after triggering
  • Prevent overlapping runs (max_active_runs)

Conclusion

While Airflow doesn’t explicitly use the term Force Start, its manual trigger functionality provides equivalent control. It allows you to run workflows instantly, making it essential for debugging, recovery, and ad-hoc processing.

Control-M Force Start Job

 

Control-M Force Start Job: Complete Guide

Introduction

Control-M is a powerful workload automation and job scheduling tool used in enterprise environments. It manages batch processing, workflows, and dependencies across systems.

In real-world operations, there are situations where you need to run a job immediately, bypassing its scheduling rules. This is done using Force Start in Control-M.


What is Force Start in Control-M?

Force Start allows you to execute a job instantly, ignoring:

  • Scheduling criteria (time/date)
  • Dependencies (predecessor jobs, conditions)
  • Resource constraints (in some cases)

It is essentially a manual override to trigger execution.


Methods to Force Start a Job

1. Using Control-M GUI (Monitoring Domain)

Steps:

  1. Open Control-M Monitoring/Active Jobs
  2. Locate the job
  3. Right-click on the job
  4. Select:
    • "Force Start" (for jobs already in Active Jobs), or
    • "Run" → "Force"

2. Using Command Line (ctmorder)

To force a job via CLI:

ctmorder -folder <folder_name> -name <job_name> -force y

Example:

ctmorder -folder DAILY_JOBS -name LOAD_CUSTOMER_DATA -force y

This:

  • Orders the job
  • Forces it to run immediately

3. Force Running a Job Already Ordered

If the job is already in the Active Jobs file:

ctmrun -job <job_name> -force y

When to Use Force Start

1. Testing & Debugging

  • Running jobs without waiting for dependencies
  • Validating fixes

2. Production Recovery

  • Upstream job failed but downstream must run
  • Restarting critical workflows

3. Urgent Business Needs

  • Immediate execution required outside schedule

Important Considerations

⚠️ Skips Dependencies

  • Job may fail if required inputs are not ready

⚠️ Data Risks

  • Running out of sequence can cause incorrect results

⚠️ Resource Conflicts

  • May ignore resource limits, leading to overload

Checking Job Status

You can monitor job execution using:

GUI:

  • Active Jobs / Monitoring domain

CLI:

ctmpsm -LISTALL

Difference: Normal Run vs Force Start

FeatureNormal RunForce Start
Respects scheduleYesNo
Checks dependenciesYesNo
Immediate executionNoYes

Best Practices

  • Use force start only when necessary
  • Verify upstream data availability
  • Monitor job closely after triggering
  • Document manual interventions

Conclusion

The Force Start feature in Control-M is essential for operational flexibility. It allows immediate execution of jobs during testing, recovery, or urgent scenarios. However, since it bypasses standard scheduling rules, it should be used carefully to avoid workflow and data issues.

Autosys Force Start Job

 

Autosys Force Start Job: A Complete Guide

Introduction

Autosys is a widely used job scheduling tool in enterprise environments. It helps automate workflows by defining, scheduling, and monitoring jobs.

Sometimes, you may need to start a job manually, bypassing its defined conditions. This is where the force start option becomes useful.


What is Force Start in Autosys?

Force start allows you to run a job immediately, regardless of:

  • Its scheduled start time
  • Dependency conditions (success/failure of other jobs)
  • Box conditions

It essentially tells Autosys:

"Run this job now, ignoring normal rules."


Command to Force Start a Job

The command used is:

sendevent -E FORCE_STARTJOB -J <job_name>

Example:

sendevent -E FORCE_STARTJOB -J data_load_job

This will immediately trigger the job data_load_job.


When to Use Force Start

Force start is useful in scenarios like:

1. Testing Jobs

  • Running a job without waiting for dependencies
  • Validating fixes or changes

2. Recovery Situations

  • When a dependent job failed but you still want to proceed
  • Restarting workflows manually

3. Urgent Execution

  • Business-critical jobs that must run immediately

Important Considerations

While force start is powerful, it should be used carefully.

⚠️ Ignores Dependencies

  • Jobs may fail if required upstream jobs haven’t completed

⚠️ Data Integrity Risks

  • Running jobs out of sequence can cause incorrect data processing

⚠️ Audit & Monitoring

  • Force-started jobs may not follow normal workflow tracking

Checking Job Status

After force starting a job, monitor it using:

autorep -J <job_name> -q

This shows:

  • Current status
  • Start time
  • Exit code

Difference Between Start and Force Start

FeatureSTARTJOBFORCE_STARTJOB
Respects scheduleYesNo
Checks dependenciesYesNo
Immediate executionNoYes

Best Practices

  • Use force start only when necessary
  • Verify dependencies manually before triggering
  • Monitor job execution closely
  • Inform stakeholders if running out of schedule

Conclusion

The FORCE_STARTJOB command in Autosys is a powerful tool for manual intervention. It enables immediate execution of jobs, especially useful during testing and recovery scenarios. However, because it bypasses standard controls, it should be used responsibly to avoid unintended consequences.

Friday, 6 March 2026

Complete beginner-friendly guide to deploy a React (UI) + Java (backend) app on Google Cloud Platform (GCP) and point your GoDaddy domain (mywebsite.com) so that when someone visits your site

 Below is a complete beginner-friendly guide to deploy a React (UI) + Java (backend) app on Google Cloud Platform (GCP) and point your GoDaddy domain (mywebsite.com) so that when someone visits your site, it loads from GCP.


Goal (what we’re building)

You already bought mywebsite.com on GoDaddy. You want:

  • React UI hosted on GCP

  • Java backend API hosted on GCP

  • Domain routing:

    • https://mywebsite.com → React UI

    • https://api.mywebsite.com → Java backend (recommended)

  • Secure HTTPS with Google-managed certificates

  • Production-ready and beginner-friendly

This guide uses Cloud Run (easiest modern way). It runs containers, scales automatically, and works great for Java.


What you need before starting

  1. A GCP account + billing enabled

  2. Your project created in GCP

  3. GoDaddy access (DNS settings)

  4. Installed tools on your machine:

    • Google Cloud SDK (gcloud)

    • Docker

    • Node.js (for React build)

    • Java + Maven/Gradle (for backend build)


Architecture options (choose one)

Option A (recommended): Cloud Run for backend + Cloud Storage/CDN for UI

  • UI is static (fast + cheap)

  • Backend is on Cloud Run

  • Best performance and standard setup

Option B: Cloud Run for both UI and backend

  • Simplest to understand (both are containers)

  • Slightly less optimized for static UI

I’ll explain Option A fully (best practice), and at the end I’ll include Option B quickly.


Part 1 — GCP project setup

1) Create/select a GCP project

In GCP Console:

  • Go to IAM & Admin → Manage resources

  • Create a project like: mywebsite-prod

2) Enable required APIs

Go to APIs & Services → Library and enable:

  • Cloud Run API

  • Artifact Registry API

  • Cloud Build API

  • Certificate Manager API (or “Cloud Managed Certificates” depending on UI)

  • Cloud DNS API (optional, not required if using GoDaddy DNS)

  • Cloud Storage API

  • (Optional but recommended) Cloud CDN, Load Balancing APIs


Part 2 — Deploy Java backend to Cloud Run

Cloud Run deploys containers, so we’ll containerize your Java backend.

3) Containerize your Java backend

In your Java backend project root, create a Dockerfile.

If you’re using Spring Boot (common)

# Build stage
FROM maven:3.9-eclipse-temurin-17 AS build
WORKDIR /app
COPY . .
RUN mvn -DskipTests package

# Run stage
FROM eclipse-temurin:17-jre
WORKDIR /app
COPY --from=build /app/target/*.jar app.jar
EXPOSE 8080
ENTRYPOINT ["java","-jar","app.jar"]

Important: Cloud Run expects your app to listen on PORT (usually 8080), so in Spring Boot make sure it runs on 8080 (default is fine).


4) Create an Artifact Registry repo (once)

In Cloud Console:

  • Artifact Registry → Repositories → Create

  • Format: Docker

  • Name: mywebsite-repo

  • Region: pick one (ex: asia-south1 / us-central1)


5) Build and push image (easy way: Cloud Build)

Open terminal and run:

gcloud config set project YOUR_PROJECT_ID
gcloud auth login
gcloud auth configure-docker

Build + push using Cloud Build:

gcloud builds submit --tag REGION-docker.pkg.dev/YOUR_PROJECT_ID/mywebsite-repo/mybackend:1.0 .

Example:
us-central1-docker.pkg.dev/mywebsite-prod/mywebsite-repo/mybackend:1.0


6) Deploy backend to Cloud Run

gcloud run deploy mybackend \
--image REGION-docker.pkg.dev/YOUR_PROJECT_ID/mywebsite-repo/mybackend:1.0 \
--region REGION \
--platform managed \
--allow-unauthenticated

After deploy, Cloud Run gives you a URL like:
https://mybackend-xxxxx-uc.a.run.app

Test:

  • Open it in browser

  • Or call a health endpoint like /actuator/health


7) Configure CORS (important for React calling backend)

If your UI will be mywebsite.com and API will be api.mywebsite.com, allow that origin.

Spring example (conceptually):

  • Allow origin: https://mywebsite.com

  • Allow methods: GET/POST/PUT/DELETE/OPTIONS

This step depends on your Java framework. Do it now so browser requests don’t fail.


Part 3 — Deploy React UI (static hosting on GCP)

React is best hosted as static files.

8) Build your React app

Inside your React project:

npm install
npm run build

This produces a build/ folder.


9) Create a Cloud Storage bucket for website hosting

Go to Cloud Storage → Buckets → Create

  • Name: mywebsite-ui-bucket (must be globally unique)

  • Location: same region or multi-region

  • Public access: we will handle properly via LB/CDN (recommended), but for simplest beginner approach you can make it public.

Simple beginner approach (public bucket hosting)

In the bucket:

  • Upload contents of build/ (not the folder itself—upload files inside it)

  • Configure website:

    • index.html

    • 404.html (or index.html for SPA routing)

SPA routing tip: React apps need unknown routes to return index.html (so /about works). We’ll handle that better with Load Balancer later.


Part 4 — Connect domain in GoDaddy (DNS) to GCP

You want mywebsite.com to go to GCP.

To support HTTPS and clean routing, best practice is:

  • Put a Global HTTPS Load Balancer in front

  • Attach:

    • UI backend (Cloud Storage bucket)

    • API backend (Cloud Run service)

  • Then map your domain to the Load Balancer IP

This sounds scary, but it’s the most “real production” setup.


10) Create a Load Balancer (UI + API under one domain)

What we want the load balancer to do

  • Requests to mywebsite.com/* → Cloud Storage (React UI)

  • Requests to api.mywebsite.com/* → Cloud Run (Java backend)

  • Google-managed SSL certificates for both

Steps (high-level)

In GCP Console:

  1. Go to Network Services → Load balancing

  2. Create HTTP(S) Load Balancer

  3. Create Frontend

    • HTTPS

    • Add domains:

      • mywebsite.com

      • www.mywebsite.com

      • api.mywebsite.com

    • Request Google-managed certificate

  4. Create Backend

    • Backend 1: Cloud Storage bucket (UI)

    • Backend 2: Cloud Run service (API) using “Serverless NEG”

  5. URL Maps / Routing:

    • Host rule mywebsite.com → UI backend

    • Host rule www.mywebsite.com → UI backend

    • Host rule api.mywebsite.com → API backend

  6. Reserve a static external IP for the LB (recommended)

After creation, GCP gives you an IP like: 34.xxx.xxx.xxx


11) Update GoDaddy DNS

Now go to GoDaddy → Domain → DNS

Add/Update records:

Root domain (mywebsite.com)

GoDaddy often supports an A record:

  • Type: A

  • Name: @

  • Value: <Load Balancer IP>

  • TTL: default

www subdomain (www.mywebsite.com)

  • Type: CNAME

  • Name: www

  • Value: mywebsite.com

API subdomain (api.mywebsite.com)

If using same LB IP (recommended with host-based routing):

  • Type: A

  • Name: api

  • Value: <Load Balancer IP>


12) Wait for DNS + SSL to become active

DNS can take minutes to hours to propagate.
SSL certificate provisioning may take some time too (commonly 15–60 minutes, sometimes longer).

Once active:

  • https://mywebsite.com loads React

  • https://api.mywebsite.com hits your backend


Part 5 — Connect React UI to Java API

13) Use environment variables in React

In React, create .env.production:

REACT_APP_API_BASE_URL=https://api.mywebsite.com

Then in code:

const API = process.env.REACT_APP_API_BASE_URL;
fetch(`${API}/your-endpoint`)

Rebuild:

npm run build

Re-upload build files to the bucket.


Part 6 — Recommended production improvements

14) Enable Cloud CDN for UI

If you front the bucket with Load Balancer, you can enable Cloud CDN for fast global caching.

15) Add backend environment config securely

Use Cloud Run environment variables:

  • DB connection string

  • secrets (prefer Secret Manager)

16) Logging + monitoring

Cloud Run logs automatically appear in:

  • Cloud Logging

  • Cloud Monitoring


Common beginner mistakes (and fixes)

1) React routes return 404

Fix: configure LB / bucket website to serve index.html for unknown paths (SPA fallback). The Load Balancer URL map can do this cleanly.

2) CORS errors

Fix: allow origin https://mywebsite.com in Java backend.

3) Backend works by URL but not by custom domain

Fix: ensure:

  • api.mywebsite.com DNS points to LB IP

  • LB host rule routes api.mywebsite.com to Cloud Run backend

  • SSL cert includes api.mywebsite.com

4) SSL stuck in “Provisioning”

Fix checklist:

  • DNS must already point correctly to LB IP

  • No conflicting records

  • Wait a bit; if still stuck, re-check domain ownership / DNS


Option B (simpler): Host both React and Java on Cloud Run

If you don’t want load balancers/buckets yet:

  • Make a single backend that serves React build too (Java serves static files)

  • Or deploy React separately as a container on Cloud Run

But: mapping a custom domain directly to Cloud Run is possible; however routing mywebsite.com and api.mywebsite.com becomes slightly less flexible than the LB approach.

end-to-end guide to host your React UI + Java backend on AWS and point mywebsite.com to AWS so visitors load your site from AWS

 Below is a beginner-friendly, end-to-end guide to host your React UI + Java backend on AWS and point mywebsite.com to AWS so visitors load your site from AWS.

I’ll use a setup that’s popular, low-cost, and easy to maintain:

  • React (UI)Amazon S3 + CloudFront (CDN)

  • Java (Backend API)AWS Elastic Beanstalk (runs your Spring Boot/JAR or WAR with minimal ops)

  • Domain + HTTPSRoute 53 + ACM certificates

  • Optional (recommended): mywebsite.com for UI, api.mywebsite.com for backend API


1) What you’re building (simple architecture)

When a user opens mywebsite.com:

  1. CloudFront serves your React static files globally (fast + HTTPS).

  2. React calls your backend at api.mywebsite.com.

  3. Elastic Beanstalk runs your Java app behind a load balancer.

This is a clean separation and avoids mixing static hosting with server APIs.


2) Prerequisites

  • An AWS account

  • Your domain: mywebsite.com (registered anywhere is fine)

  • React project builds correctly locally (npm run build)

  • Java backend packaged (commonly Spring Boot JAR)


3) Deploy the React UI to S3 + CloudFront

Step 3.1 — Build your React app

From your React project folder:

npm install
npm run build

This creates a build/ directory (or dist/ depending on your toolchain).

Step 3.2 — Create an S3 bucket for hosting

In AWS Console → S3 → Create bucket:

  • Bucket name: something like mywebsite-ui-prod-<unique>

  • Region: choose one (any is fine)

  • Block all public access: keep it ON (recommended)

Why keep it private? Because CloudFront can securely access it while the bucket stays private.

Step 3.3 — Upload the build output to S3

You can upload via Console or CLI.

CLI method (recommended):

aws s3 sync build/ s3://YOUR_BUCKET_NAME --delete

Step 3.4 — Create a CloudFront distribution in front of S3

AWS Console → CloudFront → Create distribution:

  • Origin domain: your S3 bucket

  • Origin access: choose Origin Access Control (OAC) (recommended) and let AWS update bucket policy

  • Default root object: index.html

Important for React SPA routing: configure CloudFront to return index.html for unknown paths (so /about works). AWS provides a prescriptive pattern for React SPA on S3 + CloudFront.


4) Deploy the Java backend to Elastic Beanstalk

Elastic Beanstalk is beginner-friendly for Java: you upload your app, and it provisions EC2 + load balancer + scaling.

Step 4.1 — Package your backend

For Spring Boot (Maven), typically:

mvn clean package

You’ll get something like:

  • target/myapp.jar

Elastic Beanstalk’s Java SE platform can run compiled JAR apps directly.

Step 4.2 — Create an Elastic Beanstalk application

AWS Console → Elastic Beanstalk → Create application:

  • Environment: Web server environment

  • Platform: Java

  • Upload your application code (your JAR/WAR)

  • Choose a sample instance type (e.g., t3/t4g small) for dev

After creation, Elastic Beanstalk will give you a URL like:
http://your-env.eba-xyz.region.elasticbeanstalk.com

Step 4.3 — Check your API works

Test:

  • https://your-env.../health or any endpoint you expose

Step 4.4 — CORS (don’t skip)

If your UI runs on https://mywebsite.com and API on https://api.mywebsite.com, you must allow CORS in your backend.

For Spring Boot, configure CORS to allow your frontend domain.


5) Add HTTPS (SSL) certificates with ACM

You’ll want HTTPS on both:

  • mywebsite.com (CloudFront)

  • api.mywebsite.com (Load balancer / EB)

Step 5.1 — Certificate for CloudFront must be in us-east-1

For CloudFront, AWS requires the ACM certificate to be requested/imported in US East (N. Virginia) us-east-1.

So:
AWS Console → Certificate Manager (ACM) → switch region to us-east-1 → Request certificate for:

  • mywebsite.com

  • www.mywebsite.com (optional but common)

Use DNS validation.

Step 5.2 — Certificate for the backend (Elastic Beanstalk LB)

For the backend load balancer, you can request the certificate in the same region where your Elastic Beanstalk environment is running (not necessarily us-east-1). CloudFront’s us-east-1 rule is the special case.

Then attach that cert to the load balancer listener (HTTPS 443). Elastic Beanstalk can manage ALB listeners via configuration, or you can adjust in EC2 Load Balancers depending on how your environment is set up.


6) Point your domain (mywebsite.com) to AWS

You have two common situations:

Option A (easiest): Use Route 53 as your DNS provider

  1. Route 53 → Hosted zones → create hosted zone for mywebsite.com

  2. Update nameservers at your domain registrar to Route 53 NS records

Then create records:

For the UI

Create an A (Alias) record:

  • Name: mywebsite.com

  • Alias to: your CloudFront distribution

Route 53 alias records are the AWS-native way to route apex domains to CloudFront.

(Optional) also:

  • www.mywebsite.com → Alias to same CloudFront distribution

For the API

Create:

  • api.mywebsite.com → Alias/CNAME to the Elastic Beanstalk load balancer DNS name (or EB CNAME)

Option B: Keep DNS at your current provider

If you don’t want Route 53, create DNS records where your DNS is hosted:

  • For UI: point www to CloudFront using CNAME (easy)

  • For apex mywebsite.com: many DNS providers support ALIAS/ANAME at apex. If not, moving DNS to Route 53 is usually simplest.


7) Make the React app call the right backend

Recommended:

  • UI: https://mywebsite.com

  • API: https://api.mywebsite.com

In React, set an environment variable:

.env.production

REACT_APP_API_BASE_URL=https://api.mywebsite.com

Build and redeploy UI after changes.


8) Production checklist (quick but important)

  • CloudFront SPA routing is configured (unknown routes → index.html)

  • HTTPS works on mywebsite.com (ACM cert in us-east-1)

  • Route 53 alias to CloudFront for apex domain

  • Backend has CORS allowing https://mywebsite.com

  • Backend uses HTTPS and you redirect HTTP→HTTPS

  • Add monitoring:

    • CloudWatch logs (EB)

    • CloudFront access logs (optional)


9) Simple CI/CD idea (optional)

Once the manual flow works, automate:

  • UI:

    • GitHub Actions: build → aws s3 sync → CloudFront invalidation

  • Backend:

    • Elastic Beanstalk: deploy new JAR on push (EB CLI or GitHub Actions)


Common beginner mistakes (and fixes)

  1. CloudFront + SSL not working for your domain

    • You likely created the ACM cert in the wrong region.

    • For CloudFront it must be us-east-1

  2. React refresh on /some-route gives 403/404

    • You need SPA routing behavior (serve index.html)

  3. UI can’t call API (CORS error)

    • Fix backend CORS config to allow your UI domain.