The spinel ferrite NiFe$_2$O$_4$ is a ferrimagnetic material with a high Curie temperature, highly promising for spintronic applications. Its magnetic and electronic properties strongly depend on the presence of structural defects, in particular the cation disorder (described by the inversion degree) and oxygen vacancies, which are very common in oxides. We performed first-principle calculations to study the interplay between these two kinds of defects, which have up-to-now mostly been considered independently, while they do coexist in real samples. We show that the complex formed by a NiOh/FeTd-cation swap and a neutral oxygen vacancy is more stable than these two isolated defects. Such complexes strongly reduce the width of the minority-spin band gap due to the creation of gap states. We propose an equation, potentially useful to analyze experimental data, which describes the dependence of the magnetization on the inversion degree and on the oxygen-vacancy content.