Why Truncate Test and Validation Data

[carlguo866]

Hi, I'm just wondering why we want to truncate the test and validation data to the length of the shortest possible line. It seems like this would result in translation of incomplete code?

TransCoder/preprocessing/src/dataset.py

Lines 261 to 264 in 1c21887

	for split in ['test', 'valid']:
	for f_type in ['functions_standalone', 'functions_class']:
	truncate_files(l.folder.joinpath(
	f'{split}{self.suffix}.{f_type}.tok') for l in self.langs)