Fine-grained Defense Against Vtable Hijacking Attack Based On GCC

VTable hijacking attack can be exploited by adversaries to corrupt the control flows in C++ program,especially prevalent in moder browsers.The targets of VTable hijacking attack mainly include vtable and vtable pointer,ie.one kind of pointers pointing to certain vtable.Defenses against Vtable Hijacking attack usually utilize static analysis to build a set of valid targets and instrument runtime checks to guarantee that all the virtual calls invoked at runtime are indeed from this set.The accuracy of the set of valid targets is one key criterion to evaluate their security.As for source code,existing defense techniques employ the class inheritance relationships together with the concrete object type of virtual calls so as to geneate as accurate set as possible,however,the sets are still loose.Hence in this paper,a new approach is propsed to build more accurate sets of valid targets.Our approach tries to refine the restrictions enforced by prior work,based on the object type and the target virtual function at call sites.With the support of incremental compilation and dynamic libraries,our approach makes a big enhancement on the effectiveness of defense against Viable hijacking attack.The main work is summarized as follows:(1)We conduct a thorough investigation on existing defense techniques against Viable hijacking attack.With the class inheritance relationships and the object type information,source-code based defense techniques can generate more accurate valid sets,compared to binary-cod ebaed defense techniques.However,these sets of valid targets still cannot fully comply with the intention of the program execution,which allows attachers to circumvent them.In this paper,the principle of existing source-code based vtable protection scheme is demonstrated in details,and the reason why the valid sets generated by existing defense techniques are not accurate enough is also discussed.(2)To overcome the shortcomings of existing defense techniques,our approach uses the object type and the target virtual calls to selectively build the sets from either valid virtual functions or valid vtables,based on the condtion whether there is new definition of virtual function by any subclass of the object type or not.In this way,the restrictions enforced by prior work are further refined by our approach.We improve the effectiveness of defenses against Viable hijacking attack.(3)Based on the proposed approach,a prototype named shrinkharder is implemented on top of GCC with the ability of supporting incremental compilation and dynamic libraries.We tested shrinkharder on Google Chrome.Experimental results show that the average size of valid sets generated by shrinkhard is 24.7,whereas the average size generated by shrinkwrap is 27.That means,shrinkharder reduced the size by 9%compared to shrinkwrap.