If I understand correctly both of these forms CAN mean that you are making an effort to do something or that you are requesting that someone else makes an effort to do or try something. However, sometimes in the quizzes there seems to be an alternate meaning for when you have just done something (example: just fell asleep) and I can never seem to remember which of the forms you use in this context and when to interpret it like this. Does anyone have any tips?
In ようにする we have 様に/ように. Xようにする expresses you are trying for a purpose or goal X. It often is used for habitual efforts.
The -oう in -oうとする is the verb's volitional form. V-oうとする expresses that the decision to do V has been made, and you are going to attempt/are in the middle of attempting V. Often, this is used when something interrupts you doing V.