The difference comes from the length of the chain. In textiles, you trace back to the spinner or dyer (3 to 5 levels). In food, you trace back to agricultural practices (5 to 8 levels for a processed product), and seasonal variability complicates matters. With equivalent product numbers, food typically costs 30 to 50% more than textiles.
Read the source article: Environmental labelling: what does it cost by sector?