My line of thought is this: the most epistemically primary thing is subjective experience, because it can be known directly, thus it is undeniably real. Due to the principle of ontological parsimony, if everything can be explained in terms of experience, there is no reason to postulate something beyond it (the physical). So the way I would formulate the hard problem would be something more like “Why does our experience contain the appearance of a physical world at all, and how are they related?”.
I guess this might not resonate with you either, if you don’t believe in phenomenal consciousness as all. Personally I have a hard time understanding physicalist reductionism, how can you say that something like the experience of redness is the same thing as some pattern of neurons firing in the brain? These are clearly very different things, and even if one is entirely dependent on the other, it doesn’t mean it’s non-existent or illusory.
What I mean by subjective experience is what you might refer to as what reality looks like from a specific viewpoint or what it appears like when observed. I’m not sure whether you’re assuming a physicalist or idealist position when you say “what we observe is the physical world”. My issue with this is that observation usually implies the existence of something which is being observed, the appearance upon observation, and possibly also an observer.
If you claim that the physical world doesn’t exist independently of observation, and is thus nothing beyond the totality of observed appearances (seems to me like a form of idealism), then what is being observed? If there is no object being observed, and the fact it it apparent from multiple perspectives is simply a consequence of the coherence of observation, where do the qualities of those appearances originate from? How come things don’t cease to exist when they’re not being observed?
If you claim that the appearances don’t exist independently of the physical world being observed (the physicalist interpretation), why does the world appear different from different perspectives? How do you explain things like hallucinations (there is no physical object being observed, but still some appearance is present)?
The reason I brought up that example is because physicalists usually deny the existence of qualia and claim they’re nothing beyond the brain processes correlated with them.