Delivery Targets

When General Wisdom sends a query to your data provider, the delivery field in the query envelope specifies how results should be returned to the platform. This page covers each supported delivery mechanism and how to configure it.

Delivery Mechanisms Summary

Mechanism	Description	Use Case
inline	Results returned in the HTTP response body	Small result sets (< 5 MB)
gw_s3_sync	Results uploaded to an S3-compatible bucket	Large result sets, batch processing
webhook	Results POSTed to a configured URL	Real-time integrations, event-driven architectures
sftp	Results uploaded via SFTP	Legacy system integration, compliance requirements
snowflake	Results inserted into Snowflake via SQL API	Analytics pipelines, data warehousing
azure_blob	Results uploaded to Azure Blob Storage	Azure-native architectures
gcs	Results uploaded to Google Cloud Storage	GCP-native architectures

S3 (gw_s3_sync)

The default delivery mechanism for large result sets. Results are serialized as JSON and uploaded to an S3-compatible bucket.

Envelope Configuration

{
  "delivery": {
    "mechanism": "gw_s3_sync",
    "s3Bucket": "gw-deliveries",
    "s3KeyPrefix": "results/2024/01/"
  }
}

Field	Type	Description
s3Bucket	string	Target bucket (falls back to provider default)
s3KeyPrefix	string	Key prefix for organizing results

Result Object Key

Results are uploaded with the key pattern:

{s3KeyPrefix}{queryId}.json

For example: results/2024/01/q-a1b2c3d4.json

Local Development with LocalStack

Use LocalStack to emulate S3 locally:

docker-compose.yml

services:
  localstack:
    image: localstack/localstack:latest
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEFAULT_REGION=us-east-1

Initialize buckets on startup:

init-aws.sh

#!/bin/bash
awslocal s3 mb s3://gw-deliveries
awslocal s3 mb s3://gw-deliveries-async

awslocal s3api put-bucket-policy --bucket gw-deliveries --policy '{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": "*",
    "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
    "Resource": ["arn:aws:s3:::gw-deliveries", "arn:aws:s3:::gw-deliveries/*"]
  }]
}'

Provider Configuration

S3_ENDPOINT=http://localhost:4566  # LocalStack for development
S3_BUCKET=gw-deliveries           # Default bucket

In production, your provider needs IAM credentials with s3:PutObject permission on the target bucket. The platform provides temporary credentials via the envelope when cross-account delivery is required.

Webhook

POST results directly to a URL. Useful for real-time processing pipelines and event-driven architectures.

Envelope Configuration

{
  "delivery": {
    "mechanism": "webhook",
    "url": "https://your-system.example.com/ingest"
  }
}

Field	Type	Description
url	string	Target URL for the POST request

Payload Format

The webhook receives a JSON POST with the following structure:

{
  "queryId": "q-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "results": [
    { "id": "record-1", "name": "Acme Corp", "country": "US" },
    { "id": "record-2", "name": "Acme Ltd", "country": "GB" }
  ]
}

Local Development with Mock Webhook

Run a mock webhook receiver to inspect deliveries during development:

docker-compose.yml

services:
  webhook:
    build: ./mock-webhook
    ports:
      - "8083:8083"

mock-webhook/server.ts

import express, { Request, Response } from "express";

const app = express();
const PORT = 8083;

interface WebhookDelivery {
  id: number;
  timestamp: string;
  queryId: string | null;
  body: unknown;
}

const deliveries: WebhookDelivery[] = [];
let nextId = 1;

app.use(express.json({ limit: "10mb" }));

app.post("/webhook", (req: Request, res: Response) => {
  const delivery: WebhookDelivery = {
    id: nextId++,
    timestamp: new Date().toISOString(),
    queryId: req.body?.queryId ?? null,
    body: req.body,
  };

  deliveries.push(delivery);
  console.log(`[webhook] Received #${delivery.id} queryId=${delivery.queryId}`);

  res.status(200).json({ received: true, id: delivery.id });
});

// View all received deliveries
app.get("/deliveries", (_req: Request, res: Response) => {
  res.json(deliveries);
});

app.get("/health", (_req: Request, res: Response) => {
  res.json({ status: "ok", deliveryCount: deliveries.length });
});

app.listen(PORT, () => {
  console.log(`Mock Webhook Receiver on port ${PORT}`);
});

Security Considerations

• Always use HTTPS for production webhook URLs
• Implement request signature verification on your webhook receiver
• Set appropriate timeouts (30 seconds recommended)
• Return a 2xx status code quickly; process results asynchronously if needed

SFTP

Upload results as JSON files to an SFTP server. Commonly used for legacy system integration or compliance requirements that mandate file-based transfer.

Envelope Configuration

{
  "delivery": {
    "mechanism": "sftp",
    "host": "sftp.example.com",
    "port": 22,
    "username": "data-delivery",
    "path": "/upload/results/"
  }
}

Field	Type	Description
host	string	SFTP server hostname
port	number	SFTP port (default 22)
username	string	Authentication username
path	string	Remote directory for uploaded files

Local Development with Mock SFTP

Use the atmoz/sftp Docker image for local testing:

docker-compose.yml

services:
  sftp:
    image: atmoz/sftp
    ports:
      - "2222:22"
    command: demo:demo:::upload

This creates an SFTP server with:

• Username: demo
• Password: demo
• Upload directory: /upload

Connect with:

sftp -P 2222 demo@localhost
# Password: demo

Authentication

In production, SFTP authentication uses SSH key pairs. Register your public key during provider onboarding. Password authentication is not supported in production.

Snowflake

Insert results directly into a Snowflake table via the Snowflake SQL API. Useful for analytics pipelines where results should be immediately queryable.

Envelope Configuration

{
  "delivery": {
    "mechanism": "snowflake",
    "account": "xy12345.us-east-1",
    "database": "OSINT_DATA",
    "schema": "RAW",
    "table": "QUERY_RESULTS",
    "warehouse": "COMPUTE_WH"
  }
}

Field	Type	Description
account	string	Snowflake account identifier
database	string	Target database
schema	string	Target schema
table	string	Target table
warehouse	string	Compute warehouse for the INSERT

How It Works

Your provider serializes results and submits an INSERT statement via the Snowflake SQL API (POST /api/v2/statements):

INSERT INTO OSINT_DATA.RAW.QUERY_RESULTS (query_id, data, ingested_at)
SELECT :queryId, PARSE_JSON(:data), CURRENT_TIMESTAMP()

The SQL API returns a statement handle that can be polled for completion status.

Local Development with Mock Snowflake

docker-compose.yml

services:
  snowflake:
    build: ./mock-snowflake
    ports:
      - "8082:8082"

The mock server accepts POST /api/v2/statements and returns success responses with a statement handle, simulating the Snowflake SQL API behavior.

Authentication

Snowflake delivery uses key-pair authentication. Your provider signs JWTs with the private key registered during setup. See the Snowflake SQL API documentation for details.

Azure Blob Storage

Upload results to Azure Blob Storage. Use the Azurite emulator for local development.

Envelope Configuration

{
  "delivery": {
    "mechanism": "azure_blob",
    "accountName": "yourstorageaccount",
    "containerName": "query-results",
    "blobPrefix": "2024/01/"
  }
}

Field	Type	Description
accountName	string	Azure storage account name
containerName	string	Blob container name
blobPrefix	string	Prefix for blob names

Local Development with Azurite

Azurite is the official Azure Storage emulator:

docker-compose.yml

services:
  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    ports:
      - "10000:10000"  # Blob service
      - "10001:10001"  # Queue service

Default connection string for local development:

DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;

Authentication

In production, Azure Blob delivery uses Managed Identity or SAS tokens. Configure the appropriate authentication method during provider onboarding.

Google Cloud Storage

Upload results to a GCS bucket. Use fake-gcs-server for local development.

Envelope Configuration

{
  "delivery": {
    "mechanism": "gcs",
    "bucket": "your-gcs-bucket",
    "prefix": "query-results/2024/"
  }
}

Field	Type	Description
bucket	string	GCS bucket name
prefix	string	Object name prefix

Local Development with fake-gcs-server

docker-compose.yml

services:
  gcs:
    image: fsouza/fake-gcs-server
    ports:
      - "4443:4443"
    command: ["-scheme", "http", "-port", "4443"]

Configure your GCS client SDK to use the emulator:

export STORAGE_EMULATOR_HOST=http://localhost:4443

Authentication

In production, GCS delivery uses service account credentials. Provide a service account JSON key during provider onboarding.

Docker Compose (All Targets)

Run all delivery targets locally for development and testing:

docker-compose.yml

services:
  localstack:
    image: localstack/localstack:latest
    ports:
      - "4566:4566"
    environment:
      - SERVICES=s3
      - DEFAULT_REGION=us-east-1
    volumes:
      - ./localstack/init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh

  webhook:
    build: ./mock-webhook
    ports:
      - "8083:8083"

  sftp:
    image: atmoz/sftp
    ports:
      - "2222:22"
    command: demo:demo:::upload

  snowflake:
    build: ./mock-snowflake
    ports:
      - "8082:8082"

  azurite:
    image: mcr.microsoft.com/azure-storage/azurite
    ports:
      - "10000:10000"
      - "10001:10001"

  gcs:
    image: fsouza/fake-gcs-server
    ports:
      - "4443:4443"
    command: ["-scheme", "http", "-port", "4443"]

Start all targets:

docker compose up -d

Choosing a Delivery Mechanism

Factor	Recommendation
Result size < 5 MB	Use inline for simplicity
Result size > 5 MB	Use gw_s3_sync or cloud storage
Real-time processing needed	Use webhook
Analytics/BI pipeline	Use snowflake
Compliance requires file transfer	Use sftp
Azure-native infrastructure	Use azure_blob
GCP-native infrastructure	Use gcs
Unknown or variable	Default to gw_s3_sync