In reality, there are two limits -- one set by the manufacturer of the aircraft, taking into account empty weight, fuel weight, passenger & luggage weight, temperature, pressure, runway surface type, length, wind speed and direction, and the legal type of flight (yep - we have graphs to follow ALL of that through!); that limit a commercial pilot doesn't dare go over for legal reasons. The other one is the limit where - all safety margins aside - it really does have a real-world effect on aircraft performance - that one you don't want to go anywhere near.
Case in point - when I used to fly the family to my Mum's place in a 4-seater light twin-engine aircraft (about 1 hour flight), I legally only needed 1.5 hours fuel - we were also pretty close to MAUW (maximum all up weight) (family would always load the kitchen sink) - so my very real decision was to respect the MAUW or go slightly over it and be able to carry around 3.5 hours fuel (enough to get there - and turn-around and come back the long way if there was a problem) (eg undercarriage stuck down (higher fuel burn) or damn sheep on the runway -- millions of them here!) or if I had to divert due to weather. In reality the plane carries it just fine - even on a single engine (although it reduced the single-engine max altitude from about 4250 to probably around 3000 feet (assuming you had the engine failure whilst you were above MAUW) (which would also mean you were probably still close to the airport of departure).
So I always erred on the side of a little extra fuel and a little over the (legal) limit.
More of an issue with what you describe is aircraft balance (although I too used to weigh everything too) - I was talking to a float plane pilot one day - who once had a problem with a very large lady who sat at the back of the small plane. He said that - basically - there's just no polite way of saying "madam, you're going to have to move because you're so fat the plane will crash if we try to take off with you sitting there)
I really don't miss those days