Memory Management Guide

Efficient memory management is critical for GPU computing performance. DotCompute provides unified memory abstractions, pooling, and zero-copy operations.

Overview

GPU computing involves complex memory hierarchies:

Host Memory (CPU): System RAM, accessible by CPU
Device Memory (GPU): VRAM, high bandwidth but separate address space
Pinned Memory: Page-locked host memory for faster transfers
Unified Memory: Shared address space (Apple Silicon, some CUDA systems)

DotCompute abstracts these complexities through IUnifiedBuffer<T> and IUnifiedMemoryManager.

Memory Allocation

Basic Allocation

using DotCompute;
using DotCompute.Abstractions;
using Microsoft.Extensions.DependencyInjection;

var host = Host.CreateDefaultBuilder(args)
    .ConfigureServices(services =>
    {
        services.AddDotComputeRuntime();
    })
    .Build();

var memoryManager = host.Services.GetRequiredService<IUnifiedMemoryManager>();

// Allocate device memory
var buffer = await memoryManager.AllocateAsync<float>(
    size: 1_000_000,
    location: MemoryLocation.Device);

// Must dispose to free memory
await using (buffer)
{
    // Use buffer...
}

Memory Locations:

Device: GPU memory (fastest for kernels)
Host: CPU memory (fastest for CPU access)
Pinned: Page-locked host memory (2-3x faster transfers)
Unified: Shared memory (Apple Silicon, CUDA managed memory)

Allocation Patterns

✅ Dispose Pattern:

await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
// Automatically freed when scope exits

✅ Explicit Cleanup:

var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
try
{
    // Use buffer...
}
finally
{
    await buffer.DisposeAsync();
}

❌ Memory Leak:

var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
// Forgot to dispose - memory leaked!

Memory Pooling

DotCompute includes a high-performance memory pool that reduces allocation overhead by 90%.

Enabling Pooling

services.AddDotComputeRuntime(options =>
{
    options.MemoryPooling.Enabled = true;
    options.MemoryPooling.MaxPoolSizeBytes = 4L * 1024 * 1024 * 1024;  // 4GB
    options.MemoryPooling.TrimInterval = TimeSpan.FromMinutes(5);
});

Pooling Behavior

First Allocation (cold):

var stopwatch = Stopwatch.StartNew();
var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
stopwatch.Stop();
// Time: ~500μs (actual GPU allocation)

Subsequent Allocation (warm):

await buffer.DisposeAsync();  // Returns to pool

stopwatch.Restart();
var buffer2 = await memoryManager.AllocateAsync<float>(1_000_000);
stopwatch.Stop();
// Time: ~45μs (from pool, 11x faster)

Performance Measured:

Operation              | Without Pool | With Pool | Speedup
-----------------------|--------------|-----------|--------
Allocate 1MB           | 500μs        | 45μs      | 11.1x
Allocate 10MB          | 2.1ms        | 48μs      | 43.8x
Allocate 100MB         | 18.3ms       | 52μs      | 352x
Allocate+Free (100x)   | 51ms         | 5.2ms     | 9.8x

Pool Size Classes

The pool uses 21 size classes for efficient allocation:

Size Class | Size Range      | Use Case
-----------|-----------------|------------------
0          | 0 - 4KB         | Small buffers
1          | 4KB - 8KB       | Kernel parameters
2          | 8KB - 16KB      | Small arrays
...
10         | 1MB - 2MB       | Medium arrays
...
15         | 32MB - 64MB     | Large datasets
...
20         | 512MB+          | Huge allocations

Allocation Strategy:

Exact size match: Use from pool
Size mismatch: Allocate next larger size class
Pool empty: Allocate new memory
Pool full: Trim oldest unused buffers

Pool Statistics

var stats = await memoryManager.GetPoolStatisticsAsync();

Console.WriteLine($"Total allocations: {stats.TotalAllocations}");
Console.WriteLine($"Pool hits: {stats.PoolHits}");
Console.WriteLine($"Pool misses: {stats.PoolMisses}");
Console.WriteLine($"Hit rate: {stats.HitRate:P1}");
Console.WriteLine($"Total memory pooled: {stats.TotalPooledBytes / (1024 * 1024)} MB");
Console.WriteLine($"Active allocations: {stats.ActiveAllocations}");

// Output:
// Total allocations: 10523
// Pool hits: 9471
// Pool misses: 1052
// Hit rate: 90.0%
// Total memory pooled: 2847 MB
// Active allocations: 143

Data Transfers

Host to Device

// Allocate host data
var hostData = new float[1_000_000];
for (int i = 0; i < hostData.Length; i++)
{
    hostData[i] = i;
}

// Allocate device buffer
await using var deviceBuffer = await memoryManager.AllocateAsync<float>(
    size: hostData.Length,
    location: MemoryLocation.Device);

// Transfer to GPU
await deviceBuffer.CopyFromAsync(hostData);

// Measured: 6 GB/s on PCIe Gen3 x16

Performance Factors:

Transfer size: Larger transfers amortize overhead
Memory type: Pinned memory is 2-3x faster
PCIe generation: Gen4 is 2x faster than Gen3
Alignment: 4KB-aligned transfers are faster

Device to Host

// Execute kernel (modifies deviceBuffer)
await orchestrator.ExecuteKernelAsync("ProcessData", new { data = deviceBuffer });

// Transfer back to CPU
var result = new float[deviceBuffer.Length];
await deviceBuffer.CopyToAsync(result);

// Measured: 6 GB/s

Optimized Transfers with Pinned Memory

// Allocate pinned memory (2-3x faster)
await using var pinnedBuffer = await memoryManager.AllocateAsync<float>(
    size: 1_000_000,
    location: MemoryLocation.Pinned);

// Copy to pinned memory
var hostData = new float[1_000_000];
await pinnedBuffer.CopyFromAsync(hostData);

// Transfer to device (faster)
await using var deviceBuffer = await memoryManager.AllocateAsync<float>(
    size: 1_000_000,
    location: MemoryLocation.Device);

await deviceBuffer.CopyFromAsync(pinnedBuffer);

// Measured: 12 GB/s (2x faster than unpinned)

Trade-offs:

Pinned memory is faster to transfer
Pinned memory reduces available system RAM
Pinned memory allocation is slower (one-time cost)
Use pinned memory for repeated transfers

Pinned Memory {#pinned-memory}

Pinned (page-locked) memory provides faster CPU-GPU transfers by preventing the operating system from paging memory to disk.

When to Use Pinned Memory

Pinned memory significantly improves transfer performance but comes with constraints:

✅ Good Use Cases:

Repeated transfers of the same buffer (amortize allocation cost)
Streaming data pipelines with continuous CPU-GPU communication
Real-time applications requiring predictable transfer latency
Large transfers where 2-3x speedup justifies overhead

❌ Poor Use Cases:

One-time transfers (allocation overhead > transfer savings)
Small buffers (<1MB) where absolute time difference is negligible
Systems with limited RAM (pinned memory reduces available system memory)

Allocation and Performance

// Pinned allocation (slower, one-time cost)
var stopwatch = Stopwatch.StartNew();
var pinnedBuffer = await memoryManager.AllocateAsync<float>(
    size: 10_000_000,
    location: MemoryLocation.Pinned);
stopwatch.Stop();
Console.WriteLine($"Pinned allocation: {stopwatch.ElapsedMilliseconds}ms");
// Time: ~15-20ms (vs ~0.5ms for regular allocation)

// Transfer performance (faster, repeated benefit)
stopwatch.Restart();
await deviceBuffer.CopyFromAsync(pinnedBuffer);
stopwatch.Stop();
Console.WriteLine($"Transfer time: {stopwatch.ElapsedMilliseconds}ms");
// Time: ~3ms (vs ~8ms unpinned) - 2.7x faster

// Bandwidth calculation
var bandwidth = (pinnedBuffer.Length * sizeof(float)) / stopwatch.Elapsed.TotalSeconds / (1024 * 1024 * 1024);
Console.WriteLine($"Bandwidth: {bandwidth:F2} GB/s");
// Bandwidth: ~16 GB/s (vs ~6 GB/s unpinned)

Best Practices

Reuse Pinned Buffers: Allocate once, transfer many times
Monitor System Memory: Don't pin more than 25% of available RAM
Profile First: Measure whether pinned memory improves your specific workload
Use for Hot Paths: Apply to frequently-transferred buffers only

Example Pipeline:

// Allocate pinned staging buffer once
await using var stagingBuffer = await memoryManager.AllocateAsync<float>(
    size: batchSize,
    location: MemoryLocation.Pinned);

await using var deviceBuffer = await memoryManager.AllocateAsync<float>(
    size: batchSize,
    location: MemoryLocation.Device);

// Reuse for multiple batches (amortize allocation cost)
for (int batch = 0; batch < 1000; batch++)
{
    // CPU writes to pinned buffer
    var hostData = LoadNextBatch(batch);
    await stagingBuffer.CopyFromAsync(hostData);

    // Fast transfer to GPU (2-3x faster than unpinned)
    await deviceBuffer.CopyFromAsync(stagingBuffer);

    // Process on GPU
    await orchestrator.ExecuteKernelAsync("ProcessBatch", new { deviceBuffer });
}

Performance: Allocation overhead amortized over 1000 batches, total pipeline 2.5x faster than unpinned.

Batch Transfers

❌ Multiple Small Transfers (inefficient):

for (int i = 0; i < 1000; i++)
{
    await deviceBuffer.CopyFromAsync(
        source: hostData,
        sourceOffset: i * 1024,
        destinationOffset: i * 1024,
        count: 1024);
}
// Time: ~50ms (overhead-dominated)
// Bandwidth: ~2 GB/s

✅ Single Large Transfer (efficient):

await deviceBuffer.CopyFromAsync(hostData);
// Time: ~8ms
// Bandwidth: ~12 GB/s (6x faster)

Zero-Copy Operations

Using Span

DotCompute kernels use Span<T> for zero-copy access:

[Kernel]
public static void ProcessData(ReadOnlySpan<float> input, Span<float> output)
{
    int idx = Kernel.ThreadId.X;
    if (idx < output.Length)
    {
        output[idx] = input[idx] * 2.0f;  // Direct memory access, no copy
    }
}

// No intermediate copies needed
await orchestrator.ExecuteKernelAsync(
    "ProcessData",
    new { input = inputBuffer, output = outputBuffer });

Benefits:

No allocation overhead
No copying between buffers
Direct hardware access
Reduced memory usage

Avoiding Unnecessary Copies

❌ Unnecessary Copy:

var hostData = new float[1_000_000];
// ... populate hostData ...

// Bad: Creates intermediate array
var deviceData = hostData.ToArray();
await deviceBuffer.CopyFromAsync(deviceData);
// Extra allocation + copy

✅ Direct Transfer:

var hostData = new float[1_000_000];
// ... populate hostData ...

await deviceBuffer.CopyFromAsync(hostData);
// Single transfer, no intermediate

Kernel Chaining

❌ Transfer Between Kernels:

await orchestrator.ExecuteKernelAsync("Kernel1", new { input, intermediate });
var hostIntermediate = new float[intermediate.Length];
await intermediate.CopyToAsync(hostIntermediate);  // Unnecessary

await orchestrator.ExecuteKernelAsync("Kernel2", new { input = hostIntermediate, output });

✅ Keep Data on GPU:

await orchestrator.ExecuteKernelAsync("Kernel1", new { input, intermediate });
await orchestrator.ExecuteKernelAsync("Kernel2", new { input = intermediate, output });
// intermediate stays on GPU, no transfer

Performance Impact:

With unnecessary transfer:  45ms total (15ms compute, 30ms transfer)
Without transfer:           15ms total (15ms compute, 0ms transfer)
Speedup: 3x

Unified Memory

Apple Silicon

Apple Silicon GPUs share system memory:

if (device.Capabilities.HasFlag(AcceleratorCapabilities.UnifiedMemory))
{
    // Allocate unified memory (zero-copy access)
    await using var buffer = await memoryManager.AllocateAsync<float>(
        size: 1_000_000,
        location: MemoryLocation.Unified);

    // CPU writes directly visible to GPU (no explicit transfer)
    var span = buffer.AsSpan();
    for (int i = 0; i < span.Length; i++)
    {
        span[i] = i;
    }

    // Execute kernel (reads CPU-written data)
    await orchestrator.ExecuteKernelAsync("ProcessData", new { data = buffer });

    // CPU can read GPU results immediately
    Console.WriteLine($"Result[0] = {span[0]}");
    // No explicit transfer needed
}

Performance:

Discrete GPU:    6ms transfer + 2ms compute = 8ms
Unified Memory:  0ms transfer + 2.5ms compute = 2.5ms
Speedup: 3.2x (for small datasets)

Trade-offs:

Zero transfer overhead
Slightly slower compute (shared memory bandwidth)
Best for small datasets with frequent CPU-GPU interaction

CUDA Managed Memory

var options = new ExecutionOptions
{
    PreferredBackend = BackendType.CUDA,
    UseMangedMemory = true
};

await using var buffer = await memoryManager.AllocateAsync<float>(
    size: 1_000_000,
    location: MemoryLocation.Unified);

// CUDA automatically migrates pages between CPU and GPU
await orchestrator.ExecuteKernelAsync("ProcessData", new { data = buffer }, options);

Automatic Migration:

Pages migrate on first access (demand paging)
Oversubscription: Larger than GPU memory supported
Automatic eviction: Least-recently-used pages
Performance: 10-20% overhead vs explicit transfers

Memory Lifecycle

Scope-Based Management

✅ Using Declaration (C# 8+):

public async Task ProcessData()
{
    await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
    // Use buffer...
}  // Automatically disposed here

✅ Using Statement:

await using (var buffer = await memoryManager.AllocateAsync<float>(1_000_000))
{
    // Use buffer...
}  // Disposed here

Long-Lived Buffers

public class DataProcessor
{
    private IUnifiedBuffer<float>? _buffer;

    public async Task InitializeAsync()
    {
        _buffer = await _memoryManager.AllocateAsync<float>(1_000_000);
    }

    public async Task ProcessAsync(float[] data)
    {
        if (_buffer == null)
            throw new InvalidOperationException("Not initialized");

        await _buffer.CopyFromAsync(data);
        await _orchestrator.ExecuteKernelAsync("Process", new { data = _buffer });
    }

    public async ValueTask DisposeAsync()
    {
        if (_buffer != null)
        {
            await _buffer.DisposeAsync();
            _buffer = null;
        }
    }
}

Buffer Reuse

✅ Reuse Same Buffer:

await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);

for (int i = 0; i < 100; i++)
{
    var data = LoadBatch(i);
    await buffer.CopyFromAsync(data);
    await orchestrator.ExecuteKernelAsync("Process", new { data = buffer });
}
// Single allocation for 100 batches

❌ Allocate Every Time:

for (int i = 0; i < 100; i++)
{
    await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
    var data = LoadBatch(i);
    await buffer.CopyFromAsync(data);
    await orchestrator.ExecuteKernelAsync("Process", new { data = buffer });
}
// 100 allocations (even with pooling, adds overhead)

Common Pitfalls

1. Memory Leaks

Problem: Forgetting to dispose buffers

// ❌ Leak
public async Task LeakMemory()
{
    var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
    // Do work...
    // Forgot to dispose - memory leaked!
}

// After 1000 calls: 4GB leaked

Solution: Always use await using

// ✅ No leak
public async Task NoLeak()
{
    await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
    // Do work...
}  // Automatically disposed

2. Excessive Allocations

Problem: Allocating in tight loops

// ❌ Allocates 1000 times
for (int i = 0; i < 1000; i++)
{
    await using var temp = await memoryManager.AllocateAsync<float>(1024);
    // Process...
}
// Time: ~50ms (allocation overhead)

Solution: Allocate once, reuse

// ✅ Single allocation
await using var temp = await memoryManager.AllocateAsync<float>(1024);
for (int i = 0; i < 1000; i++)
{
    // Process...
}
// Time: ~5ms (10x faster)

3. Premature Disposal

Problem: Disposing buffer still in use

// ❌ Use-after-free
var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
var task = orchestrator.ExecuteKernelAsync("Process", new { data = buffer });
await buffer.DisposeAsync();  // Freed while kernel running!
await task;  // Undefined behavior

Solution: Dispose after completion

// ✅ Proper ordering
await using var buffer = await memoryManager.AllocateAsync<float>(1_000_000);
await orchestrator.ExecuteKernelAsync("Process", new { data = buffer });
// Kernel complete, safe to dispose

4. Buffer Size Mismatch

Problem: Allocating wrong size

// ❌ Buffer too small
var data = new float[1_000_000];
await using var buffer = await memoryManager.AllocateAsync<float>(100_000);  // 10x too small!
await buffer.CopyFromAsync(data);  // Throws IndexOutOfRangeException

Solution: Match sizes

// ✅ Correct size
var data = new float[1_000_000];
await using var buffer = await memoryManager.AllocateAsync<float>(data.Length);
await buffer.CopyFromAsync(data);

5. Unnecessary Pinned Memory

Problem: Over-using pinned memory

// ❌ Pins 4GB of system RAM
await using var huge = await memoryManager.AllocateAsync<float>(
    size: 1_000_000_000,
    location: MemoryLocation.Pinned);
// Reduces available system memory significantly

Solution: Use pinned memory only for repeated transfers

// ✅ Pinned only for hot path
if (transferCount > 10)  // Repeated transfers
{
    pinnedBuffer = await memoryManager.AllocateAsync<float>(
        size: 1_000_000,
        location: MemoryLocation.Pinned);
}
else  // One-time transfer
{
    regularBuffer = await memoryManager.AllocateAsync<float>(
        size: 1_000_000,
        location: MemoryLocation.Host);
}

Performance Optimization

1. Batch Small Allocations

❌ Many Small Buffers:

var buffers = new IUnifiedBuffer<float>[100];
for (int i = 0; i < 100; i++)
{
    buffers[i] = await memoryManager.AllocateAsync<float>(1024);
}
// Time: ~50ms
// Memory overhead: ~200KB (metadata)

✅ Single Large Buffer:

await using var buffer = await memoryManager.AllocateAsync<float>(100 * 1024);
// Time: ~0.5ms (100x faster)
// Memory overhead: ~2KB

2. Align Transfers

❌ Unaligned Transfer:

await buffer.CopyFromAsync(
    source: hostData,
    sourceOffset: 123,      // Unaligned
    destinationOffset: 456, // Unaligned
    count: 10007);
// Bandwidth: ~4 GB/s

✅ Aligned Transfer:

await buffer.CopyFromAsync(
    source: hostData,
    sourceOffset: 0,        // 4KB aligned
    destinationOffset: 0,   // 4KB aligned
    count: 10240);          // Multiple of 256
// Bandwidth: ~12 GB/s (3x faster)

3. Minimize Synchronization

❌ Sync After Each Transfer:

for (int i = 0; i < 10; i++)
{
    await buffer.CopyFromAsync(data[i]);
    await orchestrator.SynchronizeDeviceAsync();  // 10-50μs overhead
}
// Total overhead: 100-500μs

✅ Batch and Sync Once:

for (int i = 0; i < 10; i++)
{
    var task = buffer.CopyFromAsync(data[i]);
    // Don't await yet
}
await orchestrator.SynchronizeDeviceAsync();  // Single sync
// Total overhead: 10-50μs (10x faster)

4. Prefer Async Operations

❌ Blocking Transfer:

buffer.CopyFrom(hostData);  // Blocks thread
// CPU idle during transfer

✅ Async Transfer:

await buffer.CopyFromAsync(hostData);  // Async
// CPU can do other work during transfer

5. Use Memory-Mapped Files for Huge Datasets

// For datasets larger than GPU memory
var options = new ExecutionOptions
{
    UseMemoryMappedFiles = true,
    ChunkSize = 256 * 1024 * 1024  // 256MB chunks
};

// Automatically streams data from disk
await orchestrator.ExecuteKernelAsync(
    "ProcessHugeDataset",
    new { data = hugeDataFilePath },
    options);

Troubleshooting

Issue: Out of Memory

Symptom: OutOfMemoryException during allocation

Diagnosis:

var available = await memoryManager.GetAvailableMemoryAsync(deviceId: 0);
var total = await memoryManager.GetTotalMemoryAsync(deviceId: 0);
var used = total - available;

Console.WriteLine($"Used: {used / (1024 * 1024)} MB");
Console.WriteLine($"Available: {available / (1024 * 1024)} MB");
Console.WriteLine($"Total: {total / (1024 * 1024)} MB");

Solutions:

Reduce batch size:

int batchSize = Math.Min(requestedSize, (int)(available * 0.8));

Enable memory pooling (automatic cleanup):

services.AddDotComputeRuntime(options =>
{
    options.MemoryPooling.Enabled = true;
    options.MemoryPooling.TrimInterval = TimeSpan.FromMinutes(1);
});

Use streaming:

// Process data in chunks
int chunkSize = 10_000_000;
for (int i = 0; i < data.Length; i += chunkSize)
{
    int size = Math.Min(chunkSize, data.Length - i);
    await ProcessChunkAsync(data[i..(i + size)]);
}

Issue: Slow Transfers

Symptom: Transfers taking longer than expected

Diagnosis:

var stopwatch = Stopwatch.StartNew();
await buffer.CopyFromAsync(hostData);
stopwatch.Stop();

var bandwidth = (hostData.Length * sizeof(float)) / stopwatch.Elapsed.TotalSeconds / (1024 * 1024 * 1024);
Console.WriteLine($"Bandwidth: {bandwidth:F2} GB/s");

// Expected: 6-12 GB/s
// If < 2 GB/s: Problem detected

Solutions:

Use pinned memory:

await using var pinnedBuffer = await memoryManager.AllocateAsync<float>(
    size: hostData.Length,
    location: MemoryLocation.Pinned);

await pinnedBuffer.CopyFromAsync(hostData);
await deviceBuffer.CopyFromAsync(pinnedBuffer);
// 2-3x faster

Check alignment:

// Ensure 4KB-aligned transfers
int alignedSize = (hostData.Length + 1023) & ~1023;

Reduce transfer frequency:

// Batch multiple small transfers into one large transfer

Issue: Memory Fragmentation

Symptom: Allocation fails despite sufficient total memory

Diagnosis:

var stats = await memoryManager.GetFragmentationStatsAsync();
Console.WriteLine($"Largest free block: {stats.LargestFreeBlock / (1024 * 1024)} MB");
Console.WriteLine($"Fragmentation: {stats.FragmentationRatio:P1}");

Solution: Defragment pool

await memoryManager.DefragmentPoolAsync();
// Compacts memory, may take 10-100ms

Platform-Specific Notes

NVIDIA CUDA

Allocation Limits:

Single allocation: Up to GPU memory size
Total allocations: Unlimited (limited by memory)
Pinned memory: Typically 25% of system RAM

Optimal Configuration:

services.AddDotComputeRuntime(options =>
{
    options.CUDA.PinnedMemoryPoolSize = 2L * 1024 * 1024 * 1024;  // 2GB
    options.CUDA.EnableManagedMemory = false;  // Explicit transfers faster
    options.MemoryPooling.MaxPoolSizeBytes = 16L * 1024 * 1024 * 1024;  // 16GB
});

Apple Metal

Unified Memory:

services.AddDotComputeRuntime(options =>
{
    options.Metal.UseSharedMemory = true;  // Zero-copy access
    options.Metal.ResourceStorageMode = ResourceStorageMode.Shared;
});

Performance:

Shared memory: 2-3x faster for small datasets
Private memory: 1.5x faster compute for large datasets

AMD OpenCL

Buffer Types:

services.AddDotComputeRuntime(options =>
{
    options.OpenCL.PreferReadWriteBuffers = true;  // Better performance
    options.OpenCL.UseZeroCopyBuffers = false;  // Limited platform support
});

Best Practices Summary

Always dispose buffers with await using
Enable memory pooling for frequent allocations
Reuse buffers instead of allocating in loops
Use pinned memory for repeated transfers
Minimize host-device transfers with kernel chaining
Prefer large transfers over many small ones
Align memory access to 4KB boundaries
Use unified memory on Apple Silicon
Monitor pool statistics to optimize configuration
Profile transfer bandwidth to detect issues

Table of Contents

Memory Management Guide

Overview

Memory Allocation

Basic Allocation

Allocation Patterns

Memory Pooling

Enabling Pooling

Pooling Behavior

Pool Size Classes

Pool Statistics

Data Transfers

Host to Device

Device to Host

Optimized Transfers with Pinned Memory

Pinned Memory {#pinned-memory}

When to Use Pinned Memory

Allocation and Performance

Best Practices

Batch Transfers

Zero-Copy Operations

Using Span

Avoiding Unnecessary Copies

Kernel Chaining

Unified Memory

Apple Silicon

CUDA Managed Memory

Memory Lifecycle

Scope-Based Management

Long-Lived Buffers

Buffer Reuse

Common Pitfalls

1. Memory Leaks

2. Excessive Allocations

3. Premature Disposal

4. Buffer Size Mismatch

5. Unnecessary Pinned Memory

Performance Optimization

1. Batch Small Allocations

2. Align Transfers

3. Minimize Synchronization

4. Prefer Async Operations

5. Use Memory-Mapped Files for Huge Datasets

Troubleshooting

Issue: Out of Memory

Issue: Slow Transfers

Issue: Memory Fragmentation

Platform-Specific Notes

NVIDIA CUDA

Apple Metal

AMD OpenCL

Best Practices Summary

Further Reading