OK, well first, I am suspect of the shift rod linkage if you pulled the gearcase, then the issue began. Did the engine shift/stay in gear OK before? If so, there is something wrong. What looks like a simple shift rod connection is actually the source of alot of problems. It is easy to have loose connections and stripped threads resulting in lost shift rod motion/partial clutch dog engagement. Also, there is a fine shift rod adjustment up top on the shift handle as well.
I’m not quite sure what you are describing, but it sounds like you have already pulled the gearcase apart unfortunately. Shifting the clutch dog smoothly/easily/no resistance is perfectly “normal” for many engines that did not have clutch dog ball detents..