[greffov]

you can't do it with the OS. just do it with PHP.

It's a bit ugly and expensive, but you can call du -s <dir_root> and parse the output (in bytes) such that you do not allow more uploading, etc. when the value exceeds a soft limit. You have to see whether it is too slow or not, either use a database to keep track of the files and their sizes (SELECT sum(size) FROM files WHERE owner=$userid; you get the point)