You can leave college for the OHL but you can't leave the OHL for college. That seems kind of stupid.
You can, it's just that, if you do it mid-season, you can't play until next season - though, really, by the time you reach college age, you've put in 2 or 3 junior seasons and likely only have 1 or 2 left, so, why wouldn't you stick it out and go to college after if you don't get drafted/signed?