I was taking a look back at the neck and head joint and it looks like I might need to add a retaining ring or some sort of flange to prevent the head from falling out. Although the servo horn is attached to the base of the swivel piece, it doesn't seem like a good idea to let it be the only piece holding up the head.
So, I'll probably increase the height of the swivel piece and add a groove around it. Then I'll add a retaining piece onto the top of the upper neck and that'll be bolted down. Hopefully, that should do the trick.