in fact, yes you can. You will have to sign a contract with him and you will need 5 to 10 black prisms to do so. When you sign the first contact (which is standard in the game) you will start a quest and when you talk to him again, he will ask you for the prisms (I just don't remember the exact number) and then you will be able to change the difficulty. There are other contracts there, but if I'm not mistaken, the only one that doesn't need prisms is the difficulty one.