ARLET: Adaptive Representation Learning with End-to-end Training