What can be done to protect the system from overflow exploits?

An overflow occurs when a program writes to the memory address which is outside the intended data structure. The data structure in such cases is usually a fixed length buffer. It results in the corruption of the data. The default behaviour of the solaris kernel is to map the system stack RWX. A non-executable stack gets in the way of the certain classes of security bug exploits. A feature was added that would enable the administrators to disable the ‘X’ protection from a stack. To enable this the user needs to add the following to the /etc/system:

- Foil certain classes of bug exploits set noexec_user_stack = 1 - Log attempted exploits set noexec_user_stack_log = 1

To be able to use this feature certain prerequisites are necessary such as hardware support, also 32 bit applications which are being run on a 64 bit kernel do not work.