For racing under UK MSA licences there is a very well-defined series of punishments for exceeding track limits:
https://www.msauk.org/assets/tracklimitsguidance-3.pdf
Those are the penalties imposed during races; in a timed practice session the time for any lap on which a track limit transgression is reported will be disallowed.
Some circuits, Oulton Park, Brands Hatch, Snetterton, have track limit sensors at points around the circuit where advantage can be gained by exceeding track limits. The sensors are linked to cameras, with a display in Race Control, so that penalties can be immediately applied by the Clerk of the Course. Elsewhere, it's down the the judgement of a trackside observer who will, or should, be appointed as a Judge of Fact - drives are not allowed to appeal the decisions of a JoF.