The fashion industry’s negative impact and overconsumption require urgent action to improve and reduce fashion consumption. Tactile gesture plays a vital role in understanding, selecting, and feeling attached to clothes. In this paper, we introduce the FabricTouch II dataset with multimodal infromation, which focuses on fabric assessment touch gestures and aims to support sustainable fashion consumption. By integrating gesture labels, we enhance the dataset’s comprehensiveness, improve recognition accuracy, and provide valuable information for consumers and intelligent systems, such as conversational agents in shop or home wardrobe. Additionally, this study has made preliminary explorations on recognizing fabric touch gestures using time-spectral representations of EMG combined with graph representations on this small batch dataset. The experiment found that the graph representation of EMG outperforms the regular neural network and that the representation capacity of bilateral EMG data is superior to that of unilateral data.