There seems to be a bug that hinders navigation of the Table of Contents of an EPUB file in the Contents Inspector. I’ve made a screen recording, but cannot upload it since I’m still a new user.
One can navigate the top level of the ToC using the up/down arrow keys, and one can expand a top level entry with the right arrow … but if you try to move into the expanded ToC entries by pressing the down arrow, the highlight jumps back to the top level entry.
Clicking on non-toplevel entries using the mouse also fails — the highlight simply jumps back to the top level.
Navigating the ToC of a PDF file works just fine for me.