Nine times out of ten, it probably goes against the grain to do any such thing. However...
I don't think we're exactly talking sprezzatura here, but something rather less 'considered' - (not the unbuttoned-button-down, the watch worn on the shirt sleeve, the collar ends allowed to project from the crew neck instead of being tucked in - see Montgomery Clift, Agnelli, John Simons and J.P.Gaul on these refinements). Throw something on? For public display - 'the display of a lack of display' - I'll probably duck out - but in private...
Sometimes I think it's possible. Right at this very moment I'm wearing a Black Watch flannel shirt, raspberry-coloured cotton cable-knit v-neck, black Wranglers, oatmeal hiking socks...
I would suggest, reexamining some of the images from 'Take Ivy' that this approach is slightly closer to the collegiate 'Boom Years' look than anything that might have been seen in England. Don't they look utterly relaxed, just as far from the igent stereotype as it's possible to imagine? I got a bit sniffy about 'Take Ivy' in around 2009 but have begun to enjoy it once again - albeit ditching the 'yellow rain slicker' look and certain other elements as not quite workable. But Madras shorts, button-downs with the sleeves rolled, long socks and loafers all resonate. Also my wife, out of the blue, reminded me of the look worn by Pete Campbell's father in 'Mad Men'.
Yet how to avoid it all being too close for comfort to 'cosplay'?