A common instrumental function is to prevent change
up:: AI Alignment MOC Changing your utility function always ranks low since it poses a thread to you being able to fulfill your terminal goal.
Think about it, in life we don’t experience continuous 100% happiness. Worries and discomfort often cloud our mind. If you were offered a pill that would bring you continuous yet would make you love to kill your kids, you wouldn’t take it. It doesn’t matter how nice it will be after the change, your current goal of loving your kids would never allow you to take this pill.
add: it will ‘volkswagen’ you. works well in the test environment but not when you don’t look or until it optimised itself until you can’t shut it down