For continued neglect of duty, drunkenness, incompetency or official misconduct, the state board of education may remove a county superintendent. However, before the removal, the officer shall have ten (10) days' notice of the charge, and shall be allowed opportunity to make defense.