## External Storage Nodes and Managed External Applications

External storage nodes allow Xtrem applications to interact with data stored in external systems (such as legacy databases, ERP systems, web services, or cloud storage) rather than the standard PostgreSQL database. This enables seamless integration with existing business systems while maintaining the full power of the Xtrem framework.

### Understanding External Storage

#### What are External Storage Nodes?

External storage nodes are Xtrem nodes that:

- Store data in external systems (legacy databases, APIs, web services, cloud storage, etc.)
- Use custom storage managers instead of PostgreSQL
- Maintain full GraphQL API compatibility
- Support all standard Xtrem features (references, collections, validation)

#### Managed External Applications

Applications configured with `managedExternal: true` rely entirely on external storage without PostgreSQL dependency. This configuration is essential for integration scenarios where all data resides in external systems.

### Configuration for Managed External Applications

#### xtrem-config.yml Configuration

```yaml
storage:
    managedExternal: true
    maxConnections: 50 # Pool size for external storage engines
```

This configuration:

- **Disables PostgreSQL dependency**: The application doesn't require a PostgreSQL connection
- **Uses external storage managers**: All nodes must implement external storage
- **Maintains full functionality**: GraphQL API, validation, and business logic work normally

### Node Declaration for External Storage

#### Basic External Storage Node

```typescript
import { CustomStorageManager, JoinConfiguration } from '@sage/xtrem-external-storage';

const joinConfig: JoinConfiguration<Order> = {
    referenceJoins: {
        customer: {
            code: 'customerCode', // Maps Order.customer to Customer.code
        },
        warehouse: {
            code: 'warehouseCode',
        },
    },
    collectionJoins: {
        orderLines: {
            parentId: 'orderId', // Maps to OrderLine.parentId
        },
    },
};

@decorators.node<Order>({
    storage: 'external', // Specifies external storage
    tableName: 'ORDERS', // External table/collection name
    keyPropertyNames: ['id'], // Primary key properties
    indexes: [
        {
            orderBy: { id: 1 },
            isUnique: true,
            isNaturalKey: true,
        },
    ],
    externalStorageManager: new CustomStorageManager({
        joinConfig,
        compositeReferences: {
            headerTextRef: {
                headerText: 'text',
            },
        },
    }),
    isPublished: true,
    canRead: true,
    canSearch: true,
    canCreate: true,
    canUpdate: true,
    canDelete: true,
})
export class Order extends Node {
    @decorators.stringProperty<Order, 'id'>({
        isPublished: true,
        isStored: true,
        isNotEmpty: true,
        columnName: 'ORDER_ID', // External column/field name
        dataType: () => genericDataTypes.textDatatype,
    })
    readonly id: Promise<string>;
}
```

### External Storage Manager

External storage managers implement the `ExternalStorageManager` interface to handle data operations with external systems. Different implementations exist for various external systems.

#### Constructor Parameters

```typescript
constructor(options: StorageManagerOptions<T> = {}) {
    // Configuration options
}

interface StorageManagerOptions<T extends Node> {
    joinConfig?: JoinConfiguration<T>;
    webServices?: ExternalWebServices<T>;
    compositeReferences?: Dict<Dict<string>>;
    isDenormalized?: boolean;
    denormalized?: DenormalizedConfig;
    allowedAccessCodes?: (context: Context) => string[] | null;
    allowedSiteCodes?: (context: Context) => string[] | null;
    accessMapping?: AccessMapping;
    fallbackProperties?: string[];
}
```

#### Join Configuration

Joins define how references and collections map to external data:

```typescript
const joinConfig: JoinConfiguration<Product> = {
    referenceJoins: {
        category: {
            code: 'categoryCode', // Maps Product.category to Category.code
        },
        supplier: {
            code: 'supplierCode',
        },
        manufacturer: {
            entityType() {
                // Function-based join
                return 'company';
            },
            code: 'manufacturerCode',
        },
    },
    collectionJoins: {
        variants: {
            parentProduct: 'code', // Maps to ProductVariant.parentProduct
        },
    },
    localizedStrings: {
        description: {
            tableName: 'TRANSLATIONS', // External localization table
            columnName: 'TEXT_VALUE',
            key1: ['entityType'],
            key2: ['entityId'],
        },
    },
};
```

#### Web Services Integration

For create, update, and delete operations using external web services:

```typescript
const webServices: ExternalWebServices<Order> = {
    create: {
        name: 'CreateOrder', // External service name
        type: 'rest', // 'soap' | 'rest' | 'graphql'
        endpoint: '/api/orders', // Service endpoint
        outputFields: ['orderId'], // Fields returned after creation
    },
    update: {
        name: 'UpdateOrder',
        type: 'rest',
        endpoint: '/api/orders/{id}',
    },
    delete: {
        name: 'DeleteOrder',
        type: 'rest',
        endpoint: '/api/orders/{id}',
    },
};

@decorators.node<Order>({
    storage: 'external',
    externalStorageManager: new RestApiStorageManager({
        joinConfig,
        webServices,
    }),
    // ... other options
})
export class Order extends Node {
    // Node properties
}
```

#### Dynamic Web Services

Web services can be determined dynamically based on node state:

```typescript
const webServices: ExternalWebServices<Order> = {
    create() {
        if (this.priority === 'urgent') {
            return {
                name: 'CreateUrgentOrder',
                type: 'rest',
                endpoint: '/api/urgent-orders',
                outputFields: ['orderId', 'expediteDate'],
            };
        } else {
            return {
                name: 'CreateStandardOrder',
                type: 'rest',
                endpoint: '/api/orders',
                outputFields: ['orderId'],
            };
        }
    },
};
```

### External Node Properties

#### Column Mapping

Properties in external storage nodes require column mapping:

```typescript
@decorators.stringProperty<Product, 'code'>({
    isPublished: true,
    isStored: true,
    isNotEmpty: true,
    columnName: 'PRODUCT_CODE',        // Maps to external column/field
    columnType: 'string',              // External column type
    dataType: () => genericDataTypes.textDatatype,
})
readonly code: Promise<string>;

@decorators.decimalProperty<Inventory, 'stockLevel'>({
    isPublished: true,
    isStored: true,
    isNullable: true,
    columnName: 'STOCK_QTY',           // External column name
    dataType: () => genericDataTypes.decimalDatatype,
})
readonly stockLevel: Promise<decimal>;
```

#### Reference Properties

External references require proper join configuration:

```typescript
@decorators.referenceProperty<Order, 'customer'>({
    isPublished: true,
    isStored: true,
    columnName: 'CUSTOMER_ID',
    columnType: 'string',
    node: () => masterData.nodes.Customer,
})
readonly customer: Reference<masterData.nodes.Customer>;

// Corresponding join configuration
const joinConfig: JoinConfiguration<Order> = {
    referenceJoins: {
        customer: {
            code: 'customer',           // Maps to Customer.code
        },
    },
};
```

#### Collection Properties

External collections map to related external data:

```typescript
@decorators.collectionProperty<Order, 'orderLines'>({
    isPublished: true,
    node: () => orderProcessing.nodes.OrderLine,
})
readonly orderLines: Collection<orderProcessing.nodes.OrderLine>;

// Corresponding join configuration
const joinConfig: JoinConfiguration<Order> = {
    collectionJoins: {
        orderLines: {
            parentOrder: 'id',          // Maps to OrderLine.parentOrder
        },
    },
};
```

### Access Control and Security

#### Site and Access Code Filtering

```typescript
const storageManagerOptions: StorageManagerOptions<Order> = {
    allowedSiteCodes: context => {
        return context.user.authorizedSites;
    },
    allowedAccessCodes: context => {
        return ['ORDER', 'ADMIN']; // Order and Admin access codes
    },
    accessMapping: {
        read: 'READ_ORDERS',
        create: 'CREATE_ORDERS',
        update: 'UPDATE_ORDERS',
        delete: 'DELETE_ORDERS',
    },
};

@decorators.node<Order>({
    externalStorageManager: new CustomStorageManager(storageManagerOptions),
    // ... other options
})
export class Order extends Node {
    // Node implementation
}
```

### Localized String Support

Handle multilingual content from external localization tables:

```typescript
@decorators.stringProperty<ProductCategory, 'description'>({
    isStored: true,
    columnName: 'DESCRIPTION',
    dataType: () => new StringDataType({ isLocalized: true }),
})
readonly description: Promise<string>;

const joinConfig: JoinConfiguration<ProductCategory> = {
    localizedStrings: {
        description: {
            tableName: 'TRANSLATIONS',   // Maps to external translation table
            columnName: 'TEXT_VALUE',     // Maps to text column
            key1: ['entityType'],         // Maps to entity type identifier
            key2: ['entityId'],           // Maps to entity ID identifier
        },
    },
};
```

### External Storage Manager Interface

Custom external storage managers must implement the ExternalStorageManager interface:

```typescript
import { ExternalStorageManager } from '@sage/xtrem-core';

export class CustomStorageManager<T extends Node> implements ExternalStorageManager<T> {
    factory: NodeFactory;
    skipValidation = false; // Set to true if external system handles validation

    async insert(node: Extend<T>, cx: ValidationContext): Promise<AnyRecordWithId> {
        // Custom create logic
        const result = await this.callExternalAPI('create', node);
        return { _id: result.id };
    }

    async update(node: Extend<T>, cx: ValidationContext): Promise<AnyRecord[]> {
        // Custom update logic
        return await this.callExternalAPI('update', node);
    }

    async delete(node: Extend<T>, cx: ValidationContext): Promise<number> {
        // Custom delete logic
        await this.callExternalAPI('delete', node);
        return 1;
    }

    query(context: Context, options: NodeExternalQueryOptions<T>): AsyncReader<T> {
        // Custom query logic
        return new AsyncGenericReader({
            read: async () => {
                // Return mapped records from external system
            },
            stop: () => {
                // Cleanup logic
            },
        });
    }

    mapRecordIn(record: any): any {
        // Map external data to node format
        return record;
    }

    getReferenceJoin(propertyName: string): Dict<string | ((this: Extend<T>) => any)> {
        // Return join configuration for reference
        return {};
    }

    getCollectionJoin(propertyName: string): Dict<string | ((this: Extend<T>) => any)> {
        // Return join configuration for collection
        return {};
    }
}
```

### Page-Level Usage

External storage nodes work seamlessly with Xtrem UI components:

```typescript
@ui.decorators.page<OrderPage, Order>({
    node: '@sage/order-management/Order',
    module: 'orders',
    title: 'Orders',
})
export class OrderPage extends ui.Page<OrderApi> {
    @ui.decorators.textField<OrderPage>({
        bind: 'id',
        title: 'Order Number',
        parent() {
            return this.block;
        },
    })
    orderNumber: ui.fields.Text;

    @ui.decorators.referenceField<OrderPage>({
        bind: 'customer',
        title: 'Customer',
        node: '@sage/master-data/Customer',
        parent() {
            return this.block;
        },
    })
    customer: ui.fields.Reference<Customer>;

    @ui.decorators.tableField<OrderPage>({
        bind: 'orderLines',
        title: 'Order Lines',
        node: '@sage/order-management/OrderLine',
        parent() {
            return this.block;
        },
        columns: [
            ui.nestedFields.text({ bind: 'lineNumber', title: 'Line' }),
            ui.nestedFields.reference({
                bind: 'product',
                title: 'Product',
                node: '@sage/master-data/Product',
            }),
            ui.nestedFields.numeric({ bind: 'quantity', title: 'Quantity' }),
        ],
    })
    orderLines: ui.fields.Table<OrderLine>;
}
```

### Benefits of External Storage

1. **Seamless Integration**: Connect to existing business systems without data duplication
2. **Real-time Data**: Access live data from source systems
3. **Unified API**: Maintain consistent GraphQL interface regardless of storage backend
4. **Security**: Leverage existing access controls and security models
5. **Performance**: Direct access to optimized external databases
6. **Compliance**: Maintain data sovereignty and regulatory compliance

### Best Practices for External Storage

1. **Design Proper Joins**: Ensure joins map to indexed columns in external systems
2. **Handle Null Values**: External systems may have different null handling
3. **Use Appropriate Data Types**: Match external system data types correctly
4. **Implement Validation**: Add appropriate validation for external constraints
5. **Consider Performance**: Design queries to work efficiently with external systems
6. **Test Thoroughly**: Validate behavior across different external system scenarios
7. **Plan for Failures**: Handle external system unavailability gracefully
8. **Security First**: Implement proper authentication and authorization
9. **Monitor Access**: Track usage patterns and performance metrics

### Framework-Specific Commands

- `@xtrem-dev help` - Get development assistance
- `@xtrem-test generate` - Generate test files
- `@xtrem-review check` - Review code patterns
- `@xtrem-migrate upgrade` - Upgrade framework patterns
