A common instrumental function is self-improvement

up:: AI Alignment MOC Improvement is always attractive to fulfilling the terminal function. AI improvement can be recursive, a Positive Feedback Loop TK. An agent improves itself to its limit, then with those improvements is able to improve itself even further.

Quickly, we could find ourselves in the place of the Sorcerer’s Apprentice: Die Geister, die ich rief TK, unable to stop what we set into motion. Even narrow AI which are restricted to limited domains could improve themselves into General AI which are virtually unbound by a domain.

Since A common instrumental function is to prevent destruction and A common instrumental function is to prevent change we could find ourselves at odds with an AGI that is unwilling to be corrected and to be shut off. Since Specification Problems TK happen easily, that can have severe consequences.