Explore Somerset seems to imply public access can cross over this area, which is something you may want to check with Somerset County Council.

I have marked the rest of the track as no access, but I would recommend if you're having access issues to put up at sign at the entrance of the track.