Optimization of activation functions in PyTorch impacts training speed and memory usage. Techniques include in-place operations, mixed precision training, batch normalization integration, and profiling execution time. Choosing efficient activations and data types enhances model performance and convergence stability.