The Ka‘ba is not the object of worship but an indicator of the qibla
Question
A non-Muslim asked us: a Muslim worships no one except God, so why then do Muslims prostrate (sajda) in the direction of the Ka‘ba? It appears that Muslims worship the Ka‘ba itself. What is the answer to this?
Answer
Sajda is performed only to the One God, Who has no partner; it is never performed to the Ka‘ba. Whoever prostrates to the Ka‘ba, Islam declares him a polytheist (mushrik). When prostrating, the face will necessarily be turned in some direction, and for this the direction of the Ka‘ba has been prescribed.
Source: Fatawa Mahmudiyya
Preview. The full text opens with JavaScript enabled.