In self-supervised learning, by enforcing to remain invariant to applied data augmentations, methods such as SimCLR and MoCo are able to reach quality on par with supervised approaches. We propose a method that mitigates augmentation invariance of representation without neither major changes in network architecture or modifications to the self-supervised training objective. We propose to use the augmentation information during the SSL training as additional guidance for the projector network. CASSLE is a method which can be directly applicable to typical joint-embedding SSL methods regardless of their objective functions.