Previous models of earthquake rupture dynamics have neglected interesting deformational properties of fault zone materials. While most current studies involving off-fault inelastic deformation employ simple brittle failure yield criteria such as the Drucker-Prager yield criterion, the material surrounding the fault plane itself, known as fault gouge, has the tendency to deform in a ductile manner accompanied by compaction. We incorporate this behavior into a new constitutive model of undrained fault gouge in a dynamic rupture model. Dynamic compaction of undrained fault gouge occurs ahead of the rupture front. This leads to an increase in pore pressure, which preweakens the fault, reducing the static friction. Subsequent dilatancy and softening of the gouge causes a reduction in pore pressure, resulting in fault restrengthening and brief slip pulses. This leads to localization of inelastic failure to a narrow shear zone. We extend the undrained gouge model to a study of self-similar rough faults. Extreme compaction and dilatancy occur at restraining and releasing bends, respectively. The consequent elevated pore pressure at restraining bends weakens the fault and allows the rupture to easily pass, while the decrease in pore pressure at releasing bends dynamically strengthens the fault and slows rupture. In comparison to other recent models, we show that the effects of fault roughness on propagation distance, slip distribution, and rupture velocity are diminished or reversed. Next, we represent large subduction zone megathrust earthquakes with a dynamic rupture model of a shallow dipping fault underlying an accretionary wedge. In previous models by our group [Ma, 2012; Ma and Hirakawa, 2013], inelastic deformation of wedge material was shown to enhance vertical uplift and potential tsunamigenesis. Here, we include a shallow region of velocity strengthening friction with a rate-and-state framework. We find that coseismic increase of the basal friction drives further inelastic wedge failure in comparison with our previous models, with the implication of larger tsunami generation.