maximuspowers/sig-autoencoder-fourier-5-simclr-mse-new

Weight-Space Autoencoder (TRANSFORMER)

This model is a weight-space autoencoder trained on neural network activation weights/signatures. It includes both an encoder (compresses weights into latent representations) and a decoder (reconstructs weights from latent codes).

Model Description

• Architecture: Transformer encoder-decoder
• Training Dataset: maximuspowers/muat-fourier-5
• Input Mode: signature
• Latent Dimension: 256

Tokenization

• Chunk Size: 1 weight values per token
• Max Tokens: 512
• Metadata: True

Training Config

• Loss Function: contrastive
• Optimizer: adam
• Learning Rate: 0.0001
• Batch Size: 8

Performance Metrics (Test Set)

• MSE: 0.296970
• MAE: 0.406934
• RMSE: 0.544949
• Cosine Similarity: 0.6125
• R² Score: 0.2901