🌌 Enterprise-grade IPFS integration with dual-provider architecture and comprehensive file management
Comprehensive NestJS module for IPFS (InterPlanetary File System) providing seamless integration with direct node access, HTTP gateways, content pinning, file uploads, and persistent storage management with MongoDB tracking.
📚 Table of Contents
✨ Quick Start
Installation
Copy npm install @hsuite/ipfs
Basic Setup
Copy import { IpfsModule } from '@hsuite/ipfs';
@Module({
imports: [
IpfsModule.forRootAsync({
imports: [ConfigModule],
useFactory: (configService: ConfigService) => ({
nodeUrl: configService.get('IPFS_NODE_URL', 'http://localhost:5001'),
gatewaysUrls: [
'https://ipfs.io/ipfs/',
'https://gateway.pinata.cloud/ipfs/',
'https://cloudflare-ipfs.com/ipfs/'
]
}),
inject: [ConfigService]
})
]
})
export class AppModule {}
Basic Usage
Copy @Injectable()
export class FileStorageService {
constructor(private ipfsService: IpfsService) {}
async uploadFile(file: Express.Multer.File, session: IAuth.ICredentials.IWeb3.IEntity) {
// Upload and pin file to IPFS
const cid = await this.ipfsService.uploadAndPin(file, session);
return { cid, ipfsUrl: `ipfs://${cid}` };
}
async retrieveFile(cid: string) {
// Get file with type detection
const result = await this.ipfsService.getFile(cid);
return result.data;
}
}
🏗️ Architecture
Dual Provider System
🖥️ Node Provider
Direct IPFS Node - HTTP client connection to IPFS daemon
Content Pinning - Persistent storage with ownership tracking
Pin Management - MongoDB persistence for pin records
Node Status - Health monitoring and network information
🌐 Gateway Provider
HTTP Gateways - Multiple gateway support with automatic fallback
High Availability - Redundant gateway access for reliability
Image Optimization - Specialized image URL generation
Metadata Processing - Base64 decoding and URL resolution
Core Services
📁 IpfsService
Content Retrieval - Get data and files from IPFS with type detection
File Upload - Stream-based file uploads with Multer integration
Pin Management - Pin/unpin content with ownership tracking
Provider Fallback - Automatic switching between node and gateway
🗄️ Pin Management
MongoDB Integration - Persistent storage of pin records
Ownership Tracking - Web3 entity association with content
Timestamp Records - Pin creation and management timestamps
Event System - Real-time notifications for IPFS operations
🔌 REST API Controller
File Upload Endpoints - Multipart file upload with validation
Content Access - RESTful content retrieval and management
Pin Operations - HTTP endpoints for pinning and unpinning
Metadata Services - Specialized metadata processing endpoints
Module Structure
Copy src/
├── ipfs.service.ts # Main IPFS orchestration service
├── ipfs.controller.ts # REST API endpoints
├── ipfs.module.ts # Module configuration
├── types/
│ ├── interfaces/
│ │ ├── ipfs.base.interface.ts # Base provider interface
│ │ ├── ipfs.node.interface.ts # Node provider interface
│ │ ├── ipfs.gateway.interface.ts # Gateway provider interface
│ │ ├── ipfs.pin.interface.ts # Pin record interface
│ │ └── ipfs.options.interface.ts # Configuration interface
│ └── models/
│ ├── ipfs.base.model.ts # Base provider implementation
│ ├── ipfs.node.model.ts # Node provider implementation
│ ├── ipfs.gateway.model.ts # Gateway provider implementation
│ └── ipfs.pin.model.ts # MongoDB pin schema
└── index.ts # Public API exports
🔧 API Reference
IpfsService
Core Methods
get(cid: string): Promise<{data: any}>
Retrieves content from IPFS by Content Identifier
Parameters : cid: string
- IPFS Content Identifier
Returns : Retrieved content data (JSON parsed if possible)
Fallback : Node provider → Gateway provider
getFile(ipfsUrl: string): Promise<{data: {buffer: Buffer, type: FileTypeResult}}>
Retrieves file with type detection
Parameters : ipfsUrl: string
- IPFS URL or CID
Returns : File buffer and detected MIME type
Fallback : Gateway provider → Node provider
pin(cid: string, owner: IAuth.ICredentials.IWeb3.IEntity): Promise<any>
Pins content to IPFS with ownership tracking
Parameters : CID and Web3 entity owner credentials
Returns : Pin operation result
Storage : MongoDB pin record creation
unpin(cid: string, owner: IAuth.ICredentials.IWeb3.IEntity): Promise<any>
Unpins content from IPFS (owner verification)
Parameters : CID and owner credentials for verification
Returns : Unpin operation result
Validation : Owner authorization check
uploadAndPin(file: Multer.File, session: IAuth.ICredentials.IWeb3.IEntity): Promise<string>
Uploads file to IPFS and pins with metadata
Parameters : Multer file object and session credentials
Returns : CID of uploaded and pinned content
Features : Stream processing, automatic pinning, metadata storage
REST API Endpoints
Content Operations
Retrieve content from IPFS by CID
File Operations
Copy GET /ipfs/file/:cid
POST /ipfs/upload
Retrieve files and upload new files to IPFS
Pin Management
Copy POST /ipfs/pin/:cid
DELETE /ipfs/unpin/:cid
Pin and unpin content with ownership verification
Copy GET /ipfs/metadata/:cid
Retrieve and process metadata with image URL resolution
Provider Interfaces
Node Provider Interface
Copy interface INode extends IBase {
client: IPFSHTTPClient;
getStatus(): Promise<any>;
pin(content: string, owner: IAuth.ICredentials.IWeb3.IEntity, broadcast: boolean): Promise<string>;
unpin(cid: string, owner: IAuth.ICredentials.IWeb3.IEntity): Promise<any>;
}
Gateway Provider Interface
Copy interface IGateway extends IBase {
getImageUrl(cid: string): string;
getMetadata(ipfsUrl: string): Promise<any>;
}
Pin Record Interface
Copy interface IPin {
cid: string; // IPFS Content Identifier
owner: string; // Wallet address of content owner
timestamp: number; // Unix timestamp of pin creation
metadata?: { // Optional file metadata
filename?: string;
mimetype?: string;
size?: number;
};
}
📖 Guides
IPFS Node Setup Guide
Configure local or remote IPFS nodes for direct integration. Comprehensive setup guide covering IPFS node installation, configuration optimization, network connectivity, clustering setup, and enterprise-grade IPFS deployment for high-availability content storage.
Gateway Configuration Guide
Set up multiple IPFS gateways for high availability access. Advanced configuration guide covering gateway load balancing, failover mechanisms, CDN integration, performance optimization, and enterprise gateway management for reliable content delivery.
File Upload Guide
Implement secure file uploads with size limits and validation. Detailed implementation guide covering file validation, size restrictions, MIME type checking, secure upload workflows, metadata management, and enterprise file handling best practices.
🎯 Examples
File Upload and Management
Copy import { IpfsService } from '@hsuite/ipfs';
import { IAuth } from '@hsuite/auth-types';
@Injectable()
export class DocumentStorageService {
constructor(private ipfsService: IpfsService) {}
async uploadDocument(
file: Express.Multer.File,
session: IAuth.ICredentials.IWeb3.IEntity
) {
try {
// Validate file type and size
const allowedTypes = ['application/pdf', 'image/jpeg', 'image/png', 'text/plain'];
if (!allowedTypes.includes(file.mimetype)) {
throw new Error('Unsupported file type');
}
if (file.size > 50 * 1024 * 1024) { // 50MB limit
throw new Error('File size exceeds 50MB limit');
}
// Upload and pin to IPFS
const cid = await this.ipfsService.uploadAndPin(file, session);
return {
success: true,
cid: cid,
ipfsUrl: `ipfs://${cid}`,
filename: file.originalname,
mimetype: file.mimetype,
size: file.size,
owner: session.walletId,
uploadedAt: new Date().toISOString()
};
} catch (error) {
throw new Error(`Document upload failed: ${error.message}`);
}
}
async retrieveDocument(cid: string) {
try {
const fileData = await this.ipfsService.getFile(`ipfs://${cid}`);
return {
buffer: fileData.data.buffer,
mimetype: fileData.data.type?.mime || 'application/octet-stream',
extension: fileData.data.type?.ext || 'bin',
size: fileData.data.buffer.length
};
} catch (error) {
throw new Error(`Document retrieval failed: ${error.message}`);
}
}
async getDocumentMetadata(cid: string) {
try {
const content = await this.ipfsService.get(cid);
return {
cid: cid,
contentType: typeof content.data,
dataSize: JSON.stringify(content.data).length,
lastAccessed: new Date().toISOString()
};
} catch (error) {
throw new Error(`Metadata retrieval failed: ${error.message}`);
}
}
}
Content Pinning Service
Copy import { IpfsService } from '@hsuite/ipfs';
import { Injectable } from '@nestjs/common';
@Injectable()
export class ContentPinningService {
constructor(private ipfsService: IpfsService) {}
async pinUserContent(
content: any,
session: IAuth.ICredentials.IWeb3.IEntity,
contentType: 'json' | 'text' | 'binary' = 'json'
) {
try {
let contentString: string;
// Process content based on type
switch (contentType) {
case 'json':
contentString = JSON.stringify(content, null, 2);
break;
case 'text':
contentString = content.toString();
break;
case 'binary':
contentString = Buffer.from(content).toString('base64');
break;
default:
contentString = JSON.stringify(content);
}
// Pin content to IPFS
const cid = await this.ipfsService.pin(contentString, session);
return {
success: true,
cid: cid,
ipfsUrl: `ipfs://${cid}`,
contentType: contentType,
size: contentString.length,
owner: session.walletId,
pinnedAt: new Date().toISOString()
};
} catch (error) {
throw new Error(`Content pinning failed: ${error.message}`);
}
}
async unpinUserContent(
cid: string,
session: IAuth.ICredentials.IWeb3.IEntity
) {
try {
const result = await this.ipfsService.unpin(cid, session);
return {
success: true,
cid: cid,
unpinnedBy: session.walletId,
unpinnedAt: new Date().toISOString()
};
} catch (error) {
throw new Error(`Content unpinning failed: ${error.message}`);
}
}
async bulkPinContent(
contents: Array<{data: any, type: string}>,
session: IAuth.ICredentials.IWeb3.IEntity
) {
const results = [];
for (const content of contents) {
try {
const result = await this.pinUserContent(
content.data,
session,
content.type as 'json' | 'text' | 'binary'
);
results.push(result);
} catch (error) {
results.push({
success: false,
error: error.message,
contentType: content.type
});
}
}
const successful = results.filter(r => r.success).length;
const failed = results.length - successful;
return {
total: contents.length,
successful: successful,
failed: failed,
results: results
};
}
}
Copy import { IpfsService } from '@hsuite/ipfs';
import { Injectable } from '@nestjs/common';
@Injectable()
export class MediaStorageService {
constructor(private ipfsService: IpfsService) {}
async uploadImage(
imageFile: Express.Multer.File,
session: IAuth.ICredentials.IWeb3.IEntity,
generateThumbnail: boolean = false
) {
try {
// Validate image file
const allowedImageTypes = [
'image/jpeg',
'image/png',
'image/gif',
'image/webp',
'image/svg+xml'
];
if (!allowedImageTypes.includes(imageFile.mimetype)) {
throw new Error('Invalid image format. Supported: JPEG, PNG, GIF, WebP, SVG');
}
if (imageFile.size > 25 * 1024 * 1024) { // 25MB limit for images
throw new Error('Image size exceeds 25MB limit');
}
// Upload original image
const originalCid = await this.ipfsService.uploadAndPin(imageFile, session);
const result = {
original: {
cid: originalCid,
ipfsUrl: `ipfs://${originalCid}`,
size: imageFile.size,
mimetype: imageFile.mimetype
}
};
// Generate thumbnail if requested
if (generateThumbnail && imageFile.mimetype !== 'image/svg+xml') {
try {
const thumbnailBuffer = await this.generateThumbnail(imageFile.buffer);
const thumbnailFile = {
...imageFile,
buffer: thumbnailBuffer,
originalname: `thumb_${imageFile.originalname}`,
size: thumbnailBuffer.length
};
const thumbnailCid = await this.ipfsService.uploadAndPin(thumbnailFile, session);
result['thumbnail'] = {
cid: thumbnailCid,
ipfsUrl: `ipfs://${thumbnailCid}`,
size: thumbnailBuffer.length,
mimetype: imageFile.mimetype
};
} catch (thumbError) {
console.warn('Thumbnail generation failed:', thumbError);
}
}
return {
success: true,
images: result,
uploadedBy: session.walletId,
uploadedAt: new Date().toISOString()
};
} catch (error) {
throw new Error(`Image upload failed: ${error.message}`);
}
}
async getImageWithMetadata(cid: string) {
try {
const [fileData, contentData] = await Promise.allSettled([
this.ipfsService.getFile(`ipfs://${cid}`),
this.ipfsService.get(cid)
]);
const result: any = { cid };
if (fileData.status === 'fulfilled') {
result.image = {
buffer: fileData.value.data.buffer,
mimetype: fileData.value.data.type?.mime || 'application/octet-stream',
extension: fileData.value.data.type?.ext || 'bin',
size: fileData.value.data.buffer.length
};
}
if (contentData.status === 'fulfilled') {
result.metadata = contentData.value.data;
}
return result;
} catch (error) {
throw new Error(`Image retrieval failed: ${error.message}`);
}
}
private async generateThumbnail(imageBuffer: Buffer): Promise<Buffer> {
// This would typically use a library like Sharp or similar
// For demo purposes, we'll just return a smaller version of the same image
// In production, implement actual thumbnail generation
return imageBuffer; // Placeholder implementation
}
async createImageGallery(
images: Express.Multer.File[],
session: IAuth.ICredentials.IWeb3.IEntity,
galleryMetadata: any = {}
) {
try {
const uploadResults = [];
// Upload all images
for (const image of images) {
try {
const result = await this.uploadImage(image, session, true);
uploadResults.push({
filename: image.originalname,
...result.images
});
} catch (error) {
uploadResults.push({
filename: image.originalname,
error: error.message
});
}
}
// Create gallery metadata
const gallery = {
title: galleryMetadata.title || 'Untitled Gallery',
description: galleryMetadata.description || '',
createdAt: new Date().toISOString(),
createdBy: session.walletId,
images: uploadResults,
totalImages: uploadResults.filter(r => !r.error).length,
failedUploads: uploadResults.filter(r => r.error).length
};
// Pin gallery metadata
const galleryCid = await this.ipfsService.pin(JSON.stringify(gallery), session);
return {
success: true,
galleryCid: galleryCid,
galleryUrl: `ipfs://${galleryCid}`,
gallery: gallery
};
} catch (error) {
throw new Error(`Gallery creation failed: ${error.message}`);
}
}
}
IPFS Gateway Service
Copy import { Injectable, OnModuleInit } from '@nestjs/common';
import { IpfsService } from '@hsuite/ipfs';
import { HttpService } from '@nestjs/axios';
@Injectable()
export class IpfsGatewayService implements OnModuleInit {
private availableGateways: string[] = [];
private healthyGateways: Set<string> = new Set();
constructor(
private ipfsService: IpfsService,
private httpService: HttpService
) {}
async onModuleInit() {
await this.initializeGateways();
this.startHealthChecks();
}
private async initializeGateways() {
// List of popular IPFS gateways
this.availableGateways = [
'https://ipfs.io/ipfs/',
'https://gateway.pinata.cloud/ipfs/',
'https://cloudflare-ipfs.com/ipfs/',
'https://dweb.link/ipfs/',
'https://ipfs.infura.io/ipfs/',
'https://ipfs.fleek.co/ipfs/'
];
await this.checkGatewayHealth();
}
private async checkGatewayHealth() {
const testCid = 'QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG'; // Hello World file
for (const gateway of this.availableGateways) {
try {
const response = await this.httpService.get(
`${gateway}${testCid}`,
{ timeout: 5000 }
).toPromise();
if (response.status === 200) {
this.healthyGateways.add(gateway);
}
} catch (error) {
this.healthyGateways.delete(gateway);
}
}
console.log(`Healthy gateways: ${this.healthyGateways.size}/${this.availableGateways.length}`);
}
private startHealthChecks() {
// Check gateway health every 5 minutes
setInterval(() => {
this.checkGatewayHealth();
}, 5 * 60 * 1000);
}
async getOptimalGatewayUrl(cid: string): Promise<string> {
if (this.healthyGateways.size === 0) {
throw new Error('No healthy IPFS gateways available');
}
// Return random healthy gateway
const healthyGatewayArray = Array.from(this.healthyGateways);
const randomGateway = healthyGatewayArray[
Math.floor(Math.random() * healthyGatewayArray.length)
];
return `${randomGateway}${cid}`;
}
async getContentWithFallback(cid: string): Promise<any> {
let lastError: Error;
// Try healthy gateways first
for (const gateway of this.healthyGateways) {
try {
const response = await this.httpService.get(
`${gateway}${cid}`,
{ timeout: 10000 }
).toPromise();
return response.data;
} catch (error) {
lastError = error;
continue;
}
}
// Fallback to IPFS service
try {
const result = await this.ipfsService.get(cid);
return result.data;
} catch (error) {
throw new Error(`Content retrieval failed: ${lastError?.message || error.message}`);
}
}
getGatewayStatus() {
return {
totalGateways: this.availableGateways.length,
healthyGateways: this.healthyGateways.size,
unhealthyGateways: this.availableGateways.length - this.healthyGateways.size,
healthyGatewaysList: Array.from(this.healthyGateways),
healthPercentage: Math.round((this.healthyGateways.size / this.availableGateways.length) * 100)
};
}
}
Event-Driven IPFS Service
Copy import { Injectable } from '@nestjs/common';
import { OnEvent, EventEmitter2 } from '@nestjs/event-emitter';
import { IpfsService } from '@hsuite/ipfs';
@Injectable()
export class IpfsEventService {
constructor(
private ipfsService: IpfsService,
private eventEmitter: EventEmitter2
) {}
@OnEvent('ipfs:write')
async handleIpfsWrite(payload: any) {
console.log('IPFS content written:', {
timestamp: new Date().toISOString(),
owner: payload.owner?.walletId,
contentSize: payload.content?.length || 0
});
// Emit analytics event
this.eventEmitter.emit('analytics:ipfs.content.created', {
action: 'content_created',
owner: payload.owner?.walletId,
timestamp: Date.now(),
size: payload.content?.length || 0
});
}
@OnEvent('ipfs:read')
async handleIpfsRead(payload: any) {
console.log('IPFS content read:', {
cid: payload.cid,
timestamp: new Date().toISOString(),
requester: payload.requester
});
// Track content access
this.eventEmitter.emit('analytics:ipfs.content.accessed', {
action: 'content_accessed',
cid: payload.cid,
requester: payload.requester,
timestamp: Date.now()
});
}
@OnEvent('ipfs:pin')
async handleIpfsPin(payload: any) {
console.log('IPFS content pinned:', {
cid: payload.cid,
owner: payload.owner,
timestamp: new Date().toISOString()
});
// Verify pin was successful
try {
await this.ipfsService.get(payload.cid);
this.eventEmitter.emit('ipfs:pin.verified', {
cid: payload.cid,
verified: true,
timestamp: Date.now()
});
} catch (error) {
this.eventEmitter.emit('ipfs:pin.failed', {
cid: payload.cid,
error: error.message,
timestamp: Date.now()
});
}
}
@OnEvent('ipfs:unpin')
async handleIpfsUnpin(payload: any) {
console.log('IPFS content unpinned:', {
cid: payload.cid,
owner: payload.owner,
timestamp: new Date().toISOString()
});
this.eventEmitter.emit('analytics:ipfs.content.unpinned', {
action: 'content_unpinned',
cid: payload.cid,
owner: payload.owner,
timestamp: Date.now()
});
}
}
🔗 Integration
Required Dependencies
Copy {
"@nestjs/axios": "^3.0.0",
"@nestjs/mongoose": "^10.0.0",
"@nestjs/event-emitter": "^2.0.0",
"@nestjs/platform-express": "^10.0.0",
"ipfs-http-client": "^60.0.0",
"file-type": "^19.0.0",
"mongoose": "^8.0.0",
"multer": "^1.4.5",
"@hsuite/auth-types": "^2.1.2",
"@hsuite/helpers": "^2.1.0"
}
Environment Variables
Copy # IPFS Node Configuration
IPFS_NODE_URL=http://localhost:5001
IPFS_API_URL=http://localhost:5001/api/v0
# IPFS Gateway URLs (comma-separated)
IPFS_GATEWAY_URLS=https://ipfs.io/ipfs/,https://gateway.pinata.cloud/ipfs/,https://cloudflare-ipfs.com/ipfs/
# MongoDB Configuration (for pin tracking)
MONGODB_URI=mongodb://localhost:27017/ipfs-pins
# File Upload Configuration
MAX_FILE_SIZE=104857600
ALLOWED_FILE_TYPES=image/jpeg,image/png,application/pdf,text/plain
# IPFS Service Configuration
IPFS_TIMEOUT=30000
IPFS_PIN_BROADCAST=true
Module Configuration
Copy import { IpfsModule } from '@hsuite/ipfs';
import { MongooseModule } from '@nestjs/mongoose';
import { EventEmitterModule } from '@nestjs/event-emitter';
@Module({
imports: [
// MongoDB for pin tracking
MongooseModule.forRoot(process.env.MONGODB_URI),
// Event system
EventEmitterModule.forRoot(),
// IPFS module
IpfsModule.forRootAsync({
imports: [ConfigModule],
useFactory: (configService: ConfigService) => ({
nodeUrl: configService.get('IPFS_NODE_URL'),
gatewaysUrls: configService.get('IPFS_GATEWAY_URLS',
'https://ipfs.io/ipfs/,https://gateway.pinata.cloud/ipfs/'
).split(','),
options: {
timeout: configService.get('IPFS_TIMEOUT', 30000),
enableBroadcast: configService.get('IPFS_PIN_BROADCAST', true)
}
}),
inject: [ConfigService]
})
]
})
export class AppModule {}
Docker IPFS Node Setup
Copy # docker-compose.yml
version: '3.8'
services:
ipfs-node:
image: ipfs/go-ipfs:latest
ports:
- "4001:4001" # IPFS swarm port
- "5001:5001" # IPFS API port
- "8080:8080" # IPFS gateway port
volumes:
- ./ipfs-data:/data/ipfs
environment:
- IPFS_PROFILE=server
restart: unless-stopped
Integration with HSuite Ecosystem
Copy // Complete integration with other HSuite modules
@Module({
imports: [
AuthModule,
IpfsModule,
SmartConfigModule,
EventEmitterModule.forRoot()
]
})
export class FileManagementModule {}
@Injectable()
export class SecureFileService {
constructor(
private ipfsService: IpfsService,
private authService: AuthService
) {}
async uploadSecureDocument(
file: Express.Multer.File,
session: IAuth.ICredentials.IWeb3.IEntity
) {
// 1. Validate user permissions
const hasPermission = await this.authService.validatePermission(
session,
'file:upload'
);
if (!hasPermission) {
throw new Error('Insufficient permissions for file upload');
}
// 2. Upload to IPFS with ownership tracking
const cid = await this.ipfsService.uploadAndPin(file, session);
// 3. Create secure access record
return {
cid: cid,
ipfsUrl: `ipfs://${cid}`,
owner: session.walletId,
uploadedAt: new Date().toISOString(),
accessUrl: `/api/files/secure/${cid}`,
permissions: ['read', 'share']
};
}
async getSecureDocument(cid: string, session: IAuth.ICredentials.IWeb3.IEntity) {
// 1. Validate access permissions
const hasAccess = await this.validateFileAccess(cid, session);
if (!hasAccess) {
throw new Error('Access denied to requested file');
}
// 2. Retrieve file from IPFS
const fileData = await this.ipfsService.getFile(`ipfs://${cid}`);
// 3. Log access event
this.eventEmitter.emit('file:accessed', {
cid: cid,
accessedBy: session.walletId,
timestamp: Date.now()
});
return fileData.data;
}
private async validateFileAccess(
cid: string,
session: IAuth.ICredentials.IWeb3.IEntity
): Promise<boolean> {
// Implementation would check ownership or shared access
// This is a simplified example
return true;
}
}
🌌 Decentralized Storage : Complete IPFS integration with enterprise-grade reliability and automatic fallback mechanisms.
📁 Advanced File Management : Stream-based uploads, type detection, thumbnail generation, and comprehensive metadata handling.
🔐 Ownership Tracking : Web3-native content ownership with MongoDB persistence and event-driven notifications.
Built with ❤️ by the HbarSuite Team
Copyright © 2024 HbarSuite. All rights reserved.